HEX
Server: Apache/2.4.68 (Debian)
System: Linux as-cs-widget-demo-us-central1 6.1.0-44-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.164-1 (2026-03-09) x86_64
User: root (0)
PHP: 8.2.32
Disabled: NONE
Upload Files
File: //lib/google-cloud-sdk/platform/gsutil/gslib/__pycache__/metrics.cpython-312.pyc
�

����
��dZddlmZddlmZddlmZddlmZddlZddlmZddl	m
Z
ddlZddlZddl
Z
ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZdd	lmZdd
lmZddlZddlmZddlmZdd
l m!Z!ddl"m#Z#ddl"m$Z$dZ%dZ&dZ'dZ(dZ)dZ*dZ+ejXd�Z-ej\j_ej\jadd��Z1dZ2idd�dd�dd �d!d"�d#d$�d%d&�d'd(�d)d*�d+d,�d-d.�d/d0�d1d2�d3d4�d5d6�d7d8�d9d:�d;d<�d=d>d?d@dAdBdCdDdEdFdGdH��Z3GdI�dJe4�Z5dK�Z6dL�Z7ejpe6dM���Z9e6					dTdN��Z:e6dO��Z;e6dP��Z<e6dQ��Z=e6dR��Z>dUdS�Z?y)Vz:Static data and helper functions for collecting user data.�)�absolute_import)�print_function)�division)�unicode_literalsN)�defaultdict��wraps)�input)�urllib)�VERSION)�Metric)�system_util)�CalculateThroughput)�HumanReadableToBytesz(https://ssl.google-analytics.com/collectzUA-36037335-16zUA-36037335-17�Command�RetryableError�
FatalError�PerformanceSummaryz.*google\.com$�~z.gsutil/analytics-uuid�DISABLEDzEvent Category�eczEvent Action�eazEvent Label�elzEvent Value�ev�Command Name�cd1�Global Options�cd2�Command-Level Options�cd3�Config�cd4�
Command Alias�cd5�Fatal Error�cd6�Parallelism Strategy�cd7�Source URL Type�cd8�Provider Types�cd9�	Timestamp�cd10�Execution Time�cm1�Retryable Errors�cm2�Is Google Corp User�cm3�cm4�cm5�cm6�cm7�cm8�cm9�cm10�cm11�cm12�cm13�cm14)�
Num Processes�Num Threads�#Number of Files/Objects Transferred�!Size of Files/Objects Transferred�Average Overall Throughput�Num Retryable Service Errors�Num Retryable Network Errors�Thread Idle Time Percent�Slowest Thread Throughput�Fastest Thread Throughput�
Disk I/O Timec���eZdZdZeefd�ZdZdZd�Z	e
efd��Ze
d��Ze
d��Ze
			dd��Ze
dd	��Ze
d
��Zd�Zd�Zed
dfd�ZGd�de�Zd�Zd�Zd�Zd�Z			dd�Zy)�MetricsCollectorz�A singleton class to handle metrics reporting to Google Analytics (GA).

  This class is not thread or process-safe, and logging directly to the
  MetricsCollector instance can only be done by a single thread.
  c�N�t�|_||_tj�|_tj�}tjjdd�}|rt}|j�}d}tjtj ��rd}ddd|d|d	d
t"d|t"d|i|_d
j't)j*�t)j,���|_g|_t3t4�|_d|_y)aInitialize a new MetricsCollector.

    This should only be invoked through the GetCollector or StartTestCollector
    functions.

    Args:
      ga_tid: The Google Analytics tracking ID to use for metrics collection.
      endpoint: The URL to send requests to.
    �GSUtil�use_test_GA_propertyr��v�1�tid�cid�t�eventr!r3z{system}/{release})�system�releaseN)�_GetTimeInMillis�
start_time�endpoint�logging�	getLogger�loggerrL�_GetCID�boto�config�getbool�_GA_TID_TESTING�_ValidateAndGetConfigValues�_GOOGLE_CORP_HOST_RE�match�socket�gethostname�
_GA_LABEL_MAP�	ga_params�format�platformrWrX�
user_agent�_metricsr�int�retryable_errors�perf_sum_params)�self�ga_tidr[rT�use_test_property�
config_values�is_corp_users       � platform/gsutil/gslib/metrics.py�__init__zMetricsCollector.__init__xs��'�(�D�O��D�M��#�#�%�D�K��
"�
"�
$�C����+�+�H�6L�M����f��4�4�6�M��L��!�!�&�"4�"4�"6�7��l�	�S�
�v�
�s��W��h����+�,�l�
�D�N�+�1�1����9J�:B�:J�:J�:L�2�N�D�O��D�M�(��,�D�� �D��Nc��
���g�
d��
�fd�}dD]\}}|||d����d�dD]\}}|||�fd����d	D]\}}|||d
����dD]G\}}tjj||�}|s�)	t|�}�
j	||f��I|dd
d���|ddd���|ddd���|ddd���|ddd���dj
t�
D�	cgc]}	dj|	d|	d���c}	��S#t
$r�
j	|�f�Y��wxYwcc}	w)z�Parses the user's config file to aggregate non-PII config values.

    Returns:
      A comma-delimited string of config values explicitly set by the user in
      key:value pairs, sorted alphabetically by key.
    �INVALIDc����	tjj||�}|r||�r�j||f�y|r�j|�f�yy#�j|�f�YyxYw�N)r`ra�	get_value�append)�section�category�
validation_fn�config_valueru�invalid_value_strings    ��rw�GetAndValidateConfigValuezOMetricsCollector._ValidateAndGetConfigValues.<locals>.GetAndValidateConfigValue�sr���?��{�{�,�,�W�h�?���M�,�7�
�
�
��,�7�
8�
�
�
�
��*>�?�
@���
?����h�(<�=�>�s�=A�A�A/))�Boto�https_validate_certificates)rN�disable_analytics_prompt)rN�
use_magicfile)rN�tab_completion_time_logsc�8�t|�j�dvS)N)�true�false)�str�lower��vals rw�<lambda>z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>�s��#�c�(�.�.�:J� 1�;2ry)r�r�r�i�))r��debug)r��http_socket_timeout)r��num_retries)r��max_retry_delay)rN�default_api_version)rN�%sliced_object_download_max_components)rN�parallel_process_count)rN�parallel_thread_count)rN�software_update_check_period)rN�tab_completion_timeout)�OAuth2�oauth2_refresh_retriesc�V��t|�j�xrt|��kSr})r��isdigitro)r��small_int_thresholds �rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>�s)���#�c�(�:J�:J�:L�;?� #�C��+>� >�;?ry))rN�resumable_threshold)rN�rsync_buffer_lines)rN�task_estimation_thresholdc�4�t|�j�Sr})r�r�r�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>�s��#�c�(�:J�:J�:Lry))rN�(parallel_composite_upload_component_size)rN�#parallel_composite_upload_threshold)rN�%sliced_object_download_component_size)rN� sliced_object_download_thresholdrN�check_hashesc�
�|dvS)N)�if_fast_else_fail�if_fast_else_skip�always�never�r�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>s
��#�E�#Fry�content_languagec�B�|j�xrt|�dkS)N�)�isalpha�lenr�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>s��#�+�+�-�"A�C��H��M�"Ary�json_api_versionc�X�|dj�dk(xr|ddj�S)NrrQrP)r�r�r�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>s(��#�a�&�,�,�.�C�"7�"M�C���G�O�O�<M�"Mry�
prefer_apic�
�|dvS)N)�json�xmlr�r�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>s	����8Nryr��token_cachec�
�|dvS)N)�file_system�	in_memoryr�r�s rwr�z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>s���:�9;ry�,z{0}:{1}rrP)	r`rar~rr�
ValueError�join�sortedrk)
rrr�r��
bool_category�small_int_category�large_int_category�data_size_categoryr��
size_in_bytesrarur�r�s
          @@@rwrdz,MetricsCollector._ValidateAndGetConfigValues�s�����M�%��?�#K���� ��)6�/2�3�	#K���(�#��#� ��);�/?�@�(�&(G�#��#� ��);�.L�N�	(G�(�#��#��[�[�*�*�7�4F�G�l�	�	K�.�|�<�-�
�
�
� 2�M�B�
C�(� ���F�G���#�A�C���#�M�O��h�'3�,N�P��h�'4�-;�<�
�8�8��AN�
�AN�v�I���V�A�Y��q�	�2��
�	����7�	K�
�
�
� 2�4H�I�
J�	K��8
s�D�0D<�D9�8D9c��tj�rytjst|�t_tjS)zDReturns the singleton MetricsCollector instance or None if disabled.N)rL�
IsDisabled�	_instance)rss rw�GetCollectorzMetricsCollector.GetCollector#s6���"�"�$�
��%�%�#3�F�#;�� ��%�%�%ryc�j�tj�tj�tjS)z6Returns whether metrics collection should be disabled.)rL�_disabled_cache�_CheckAndSetDisabledCacher�ryrwr�zMetricsCollector.IsDisabled-s(���'�'�/��0�0�2��+�+�+ryc�h�tjjd�dk(rd|_y	tjjd�dk(rd|_|j	�y	t
jjddd�rd|_y	tj�r&tjjd�|_y	tjjt�r5tt�5}|j�tk(|_d	d	d	�y	d|_y	#1swYy	xYw)
z1Sets _disabled_cache based on user opt-in or out.�GSUTIL_TEST_ANALYTICSrRT�2FrN�use_gcloud_storage�GA_CIDN)�os�environ�getr��StartTestCollectorr`rarbr�InvokedViaCloudSdk�path�exists�_UUID_FILE_PATH�open�read�_DISABLED_TEXT)�cls�fs  rwr�z*MetricsCollector._CheckAndSetDisabledCache4s���
�z�z�~�~�-�.�#�5� �c��
�����/�	0�C�	7�!�c��	����
���	�	�X�';�U�	C� �c��	�	'�	'�	)� "�
�
���x� 8�8�c��	������	(��� �A� �v�v�x�>�9���!� �!�c��!� �s�:D(�(D1c��|j�rdtjd<d|_|t|�|_|�ddd�}||j
_||j
_tjddk7rd	|j
_yy)
aNReset the singleton MetricsCollector with testing parameters.

    Should only be used for tests, where we want to change the default
    parameters.

    Args:
      endpoint: str, URL to post to
      user_agent: str, User-Agent string for header.
      ga_params: A list of two-dimensional string tuples to send as parameters.
    �0r�FN�b�d)�a�cr�r)	r�r�r�r�rcr�rjrmrZ)r�r[rmrjs    rwr�z#MetricsCollector.StartTestCollectorLs~�� �~�~��,/�b�j�j�(�)��C�����2�C�M����#�&�i�'�C�M�M��)�C�M�M��	�z�z�)�*�c�1�!"�c�m�m��2ryc�F�dtjd<d|_||_y)z�Reset the MetricsCollector with default parameters after testing.

    Args:
      original_instance: The original instance of the MetricsCollector so we can
        set the collector back to its original state.
    rRr�N)r�r�r�r�)r��original_instances  rw�StopTestCollectorz"MetricsCollector.StopTestCollectorgs"��+.�B�J�J�&�'��C��%�C�Mryc��tjjt�r,t	t�5}|j�}ddd�r|Stjjd�S#1swY�,xYw)z�Gets the client id from the UUID file or the SDK opt-in, or returns None.

    Returns:
      str, The hex string of the client id.
    Nr�)r�r�r�r�r�r�r�r�)r�rTs  rwr_zMetricsCollector._GetCIDssT��
�w�w�~�~�o�&��� �A��f�f�h��!�	��
��:�:�>�>�(�#�#�!� �s�A/�/A8c�:�|jj|�y)aExtends self.ga_params to include new parameters.

    This is only used to record parameters that are sent with every event type,
    such as global and command-level options.

    Args:
      new_params: A dictionary of key-value parameters to send.
    N)rj�update)rr�
new_paramss  rw�ExtendGAParamszMetricsCollector.ExtendGAParams�s��	�N�N���*�%ryc�F�|jjt|�S)z�Convenience function for getting a ga_param of the collector.

    Args:
      param_name: The descriptive name of the param (e.g. 'Command Name'). Must
        be a key in _GA_LABEL_MAP.

    Returns:
      The GA parameter specified, or None.
    )rjr�ri)rr�
param_names  rw�
GetGAParamzMetricsCollector.GetGAParam�s���>�>���m�J�7�8�8ryrc��d|fd|fd|fd|ftdt�fg}|jtj|�D��	cgc]\}}	|	��	||	f��c}	}�|jtj|j
�D��	cgc]\}}	|	��	||	f��c}	}�|�t�|jz
}|jtd|f�tjjt|��}
|jjt|jd|
|j�	��ycc}	}wcc}	}w)
a�Adds a GA metric with the given parameters to the metrics queue.

    Args:
      category: str, the GA Event category.
      action: str, the GA Event action.
      label: str, the GA Event label.
      value: int, the GA Event value.
      execution_time: int, the execution time to record in ms.
      **custom_params: A dictionary of key, value pairs containing custom
          metrics and dimensions to send with the GA Event.
    rrrrr-Nr/�POST)r[�method�bodyrm)rirY�extend�six�	iteritemsrjrZrr�parse�	urlencoder�rnr
r[rm)rrr��action�label�value�execution_time�
custom_params�params�krQ�datas           rw�CollectGAMetricz MetricsCollector.CollectGAMetric�s.��$�X���v���u�
��e�}��[�)�+;�+=�>�@�F�
�M�M��=�=��7��7�4�1�a�1�=��A��7����M�M��=�=����8��8�4�1�a�A�M��A��8���
��'�)�D�O�O�;�n�
�M�M�=�!1�2�N�C�D��<�<�!�!�&��.�1�D��M�M�����
�
����/�/�	+�,����s�
D;
�D;
�

E
�E
c�,�eZdZdZd�ZGd�de�Zy)�)MetricsCollector._PeformanceSummaryParamszEThis class contains information to create a PerformanceSummary event.c��d|_d|_d|_d|_t	�|_tjrt
j�|_	d|_
d|_d|_d|_
t|j�|_d|_d|_d|_d|_d|_d|_d|_d|_d|_y)NrF)�
num_processes�num_threads�num_retryable_service_errors�num_retryable_network_errors�set�provider_typesr�IS_LINUX�GetDiskCounters�disk_counters_start�uses_fan�
uses_slice�thread_idle_time�thread_execution_timer�_ThreadThroughputInformation�thread_throughputs�avg_throughput�total_elapsed_time�total_bytes_transferred�num_objects_transferred�is_daisy_chain�has_file_dst�
has_cloud_dst�has_file_src�
has_cloud_src�rrs rwrxz2MetricsCollector._PeformanceSummaryParams.__init__�s����d���d��*+�d�'�*+�d�'��E�d��
�	�	�#.�#>�#>�#@�� ��d�m��d�o� �d��#$�d� �!,�D�,M�,M� N�d��!�d�� $�d��%)�d�"�%&�d�"�"�d���d�� �d���d�� �d�ryc�(�eZdZdZd�Zd�Zd�Zd�Zy)�FMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformationzDA class to keep track of throughput information for a single thread.c�<�d|_d|_d|_d|_y)Nr)rr�task_start_time�	task_sizer s rwrxzOMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.__init__�s%��'(��$�"#��� $�����ryc� �||_||_yr})r$r%)rrrZ�bytes_to_transfers   rw�LogTaskStartzSMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.LogTaskStart�s��)���*��ryc��|xj||jz
z
c_|xj|jz
c_d|_d|_yr})rr$rr%)rr�end_times  rw�
LogTaskEndzQMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.LogTaskEndsB�����8�d�.B�.B�#B�B���$�$����6�$�#�����ryc�B�t|j|j�Sr})rrrr s rw�
GetThroughputzTMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.GetThroughputs ��"�4�#?�#?�#'�#:�#:�<�	<ryN)�__name__�
__module__�__qualname__�__doc__rxr(r+r-r�ryrwrr"�s��P�	�+��<ryrN)r.r/r0r1rx�objectrr�ryrw�_PeformanceSummaryParamsr�s��O�%!�N<�v�<ryr3c�l�|jd�dvry|j�|j�|_d|vr|j|d��yt	j
|�D]�\}}|dvr/t
|j|�}|st|j||�|dvr0t
|j|�}t|j|||z�|dvr2t
|j|�}||krt|j||�|d	k(s��|jjj|���y)
agUpdates the _PeformanceSummaryParams object.

    Args:
      params: A dictionary of keyword arguments.
        - uses_fan: True if the command uses fan parallelism.
        - uses_slice: True if the command uses slice parallelism.
        - avg_throughput: The average throughput of the data transfer.
        - is_daisy_chain: True if the transfer uses the daisy-chain method.
        - has_file_dst: True if the transfer's destination is a file URL.
        - has_cloud_dst: True if the transfer's destination is in the cloud.
        - has_file_src: True if the transfer has a file URL as a source.
        - has_cloud_src: True if the transfer has a cloud URL as a source.
        - total_elapsed_time: The total amount of time spent on Apply.
        - total_bytes_transferred: The total number of bytes transferred.
        - thread_idle_time: The additional amount of time that threads spent
                            idle in Apply.
        - thread_execution_time: The additional amount of time that threads
                                 spent executing in Apply.
        - num_retryable_service_errors: The additional number of retryable
                                        service errors that occurred.
        - num_retryable_network_errors: The additional number of retryable
                                        network errors that occurred.
        - num_processes: The number of processes used in a call to Apply.
        - num_threads: The number of threads used in a call to Apply.
        - num_objects_transferred: The total number of objects transferred, as
                                   specified by a ProducerThreadMessage.
        - provider_types: A list of additional provider types used.
        - file_message: A FileMessage used to calculate thread throughput and
                        number of objects transferred in the non-parallel case.
    r)�cp�rsyncN�file_message)r7)rrrrrrrrrrr)rrr
r)rr	r
)
r�rqr3�_ProcessFileMessager�r��getattr�setattrr
r�)rrrr��param�	cur_values     rw�UpdatePerformanceSummaryParamsz/MetricsCollector.UpdatePerformanceSummaryParams
s@��>���~�&�o�=�����#�!�:�:�<�d��
���
���F�>�,B��C�� �]�]�6�2��
�E�
�N�
N��D�0�0�*�=�	��
�$�&�&�
�E�
:�
�8�
8��D�0�0�*�=�	���$�$�j�)�e�2C�D�
�7�	7��D�0�0�*�=�	��u��
�$�&�&�
�E�
:�
�'�	'����+�+�2�2�5�9�73ryc��|jj|j|jf}|jrg|jj
s5|jjs|jxjdz
c_|j|j�y|j|j|j�y)aTProcesses FileMessages for thread throughput calculations.

    Update a thread's throughput based on the FileMessage, which marks the start
    or end of a file or component transfer. The FileMessage provides the number
    of bytes transferred as well as start and end time.

    Args:
      file_message: The FileMessage to process.
    rPN)rqr�
process_id�	thread_id�finishedrrrr+�timer(�size)rrr7�thread_infos   rwr8z$MetricsCollector._ProcessFileMessageRs����'�'�:�:�����!7�!7�<9�:�K�����"�"�-�-��1E�1E�1N�1N����4�4��9�4����\�.�.�/����|�0�0�,�2C�2C�Dryc	��|jd�}|rC|jdt|d�tdt	|j
j
��i��tj|j
�D]'\}}|jdt|d�td|i���)|jd�}|r|jt|��yy)zDAggregates command and error info and adds them to the metrics list.r)r�r�r1r%Nr�)r�r�_GA_COMMANDS_CATEGORYri�sumrp�valuesr�r��_GA_ERRORRETRY_CATEGORY�_GA_ERRORFATAL_CATEGORY)rr�command_name�
error_type�
num_errors�fatal_error_types     rw�_CollectCommandAndErrorMetricsz/MetricsCollector._CollectCommandAndErrorMetricshs����?�?�>�2�L���d����$9�".�� -�-?�@�#&�t�'<�'<�'C�'C�'E�#F���#&�-�-��0E�0E�"F��
�J��d���N�$;�",�N�+�,>�?��L�N�#G����}�5���
���$;�"2��4�ryc
���|j�yi}dD]%\}}t|j|�|t|<�'tjr�|jj
}tj�}t|j�D�cgc]
}|d|dz��c}�t|j�D�cgc]
}|d|dz��c}�z
|td<|jjr|jjrdnd}nd}||td	<|jjr|jjrdnd
}n|jjrdnd}||td
<|jj|jjz}	|	r5t|jj�t|	�z|td<|jj ri|jj j�D�
cgc]}
|
j#���}}
t%|�|td<t'|�|td<dj)t+|jj,��|td<|jjxr|jj.|jjxr|jj0|jj2|jjxr|jj.|jjxr|jj0d�}dj)t+t5j6|�D�
�cgc]	\}
}|r|
��c}}
��}t9|jj:�}|j<dt>||d�|��ycc}wcc}wcc}
wcc}}
w)zCAggregates PerformanceSummary info and adds the metric to the list.N))rr@)r	rA)r
rE)rrF)rrD)rrB)rrC��rJ�both�cloud�filer)�fan�slice�noner'rGrHrIr�r+)�CloudToCloud�CloudToFile�
DaisyChain�FileToCloud�
FileToFile)r�r�r�r�) rqr9rirrrrrGrHrrrrrr�floatrr-�min�maxr�r�r
rrrr�r�rYrr�_GA_PERFSUM_CATEGORY)rrr�	attr_namer��
disk_start�disk_end�stat�src_url_type�strategy�
total_time�thread�throughputs�transfer_types�
transfer_type�condr��apply_execution_times                 rw� _CollectPerformanceSummaryMetricz1MetricsCollector._CollectPerformanceSummaryMetric�s������#���M���	�5�-4�D�4H�4H�4=�-?�m�M�%�(�)������'�'�;�;�j��,�,�.�h��X�_�_�->�?�->�T�t�A�w��a�� �->�?�
@�

�Z�->�->�-@�A�-@�T�t�A�w��a�� �-@�A�
B�C��M�/�2�3�
���)�)�#�3�3�@�@�V�g�l��l�6B�M�-� 1�2�3����$�$��/�/�:�:���h� �0�0�;�;���h�;C�M�-� 6�7�8��&�&�7�7��&�&�<�<�=�J��
��$�$�5�5�
6��z�9J�
J��M�"<�=�>����.�.��,�,�?�?�F�F�H��H�f��
�
�
 �H���CF�
�C�m�M�"=�>�?�BE�
�C�m�M�"=�>�?�69�X�X��t�#�#�2�2�3�65�M�-� 0�1�2�
� � �.�.�
3��$�$�2�2�� � �.�.�
2��$�$�1�1�� � �/�/�� � �-�-�
3��$�$�2�2�� � �-�-�
2��$�$�1�1��N� �X�X��'*�}�}�^�'D�
�'D�#�
�t��
�'D�
�	��F�,����/�/�1���D���*�"6� &�(<�*�)�*��S@��A��2��@
s�O�:O
�O"�O'c��|j�|j�|jsy|s|jj	�}t
jjd�dk(rtj}tjd��}tj|j�}|5tj |j|�ddd�tj"|j�g|_|�tjd|z�}tjdj%|||��}t&j(d|g}t
jj+�}t
j,j/t&j0�|d	<t3�}	tj4|�D]0\}
}tj|�|	tj|
�<�2	t7j8||	tj:xrt<j>�
�}|jj#d�|r,|jA�|jj#d�yy#1swY���xYw#tB$rF|jj#d
�	tjD|j�Yy#YYyxYwwxYw)aZReports the collected metrics using a separate async process.

    Args:
      wait_for_report: bool, True if the main process should wait for the
        subprocess to exit for testing purposes.
      log_level: int, The subprocess logger's level of debugging for testing
        purposes.
      log_file_path: str, The file that the metrics_reporter module should
        write its logs to. If not supplied, the metrics_reporter module will
        use a predetermined default path. This parameter is intended for use
        by tests that need to evaluate the contents of the file at this path.
    Nr�r�F)�deletezr"%s"z_from gslib.metrics_reporter import ReportMetrics; ReportMetrics(r"{0}", {1}, log_file_path={2})z-c�
PYTHONPATH)�env�shellz$Metrics reporting process started...z#Metrics reporting process finished.z*Metrics reporting process failed to start.)#rOrornr^�getEffectiveLevelr�r�r�r\�WARN�tempfile�NamedTemporaryFiler��
ensure_str�name�pickle�dumpr�rk�sys�
executable�copy�pathsepr�r��dictr��
subprocess�Popen�PY3r�
IS_WINDOWS�communicate�OSError�unlink)
rr�wait_for_report�	log_level�
log_file_path�temp_metrics_file�temp_metrics_file_name�reporting_code�execution_args�exec_env�sm_envrrQ�ps
             rw�
ReportMetricszMetricsCollector.ReportMetrics�s$�� 	�'�'�)��)�)�+��=�=����+�+�/�/�1�i�	�z�z�~�~�-�.�#�5��,�,�i� �3�3�5�A�� �^�^�,=�,B�,B�C��	��k�k�$�-�-�!2�3�
��M�M�$�-�-� ��D�M�� ��n�n�W�}�%<�=�m��^�^�	8�8>��"�I�}�9>�?�N��n�n�d�N�;�N��z�z��� �H��Z�Z�_�_�S�X�X�6�H�\��
�V�F��
�
�h�'���1�"%�.�.��"3�f�S�^�^�A�
��(�
��
�
�>�%�"%�'�'�"D�k�.D�.D�G�a��k�k���>�?�	�	
�
�
�������?�@�	
�;
�	��D�
��k�k���D�E�
�
�	�	�#�(�(�)��
���
�s1�3!I8�6BJ�8J�$K�*K�K�
K�K)zhttps://example.comzuser-agent-007Nr})FNN)r.r/r0r1�_GA_TID�_GA_ENDPOINTrxr�r�rd�staticmethodr�r��classmethodr�r�r�r_r�r�rrr2r3r=r8rOror�r�ryrwrLrLqs����$�l�6 �p�)��/�m�^�!�&��&��,��,��!��!�.�"7�$4�#'�#��#�4�	&��	&��$��$� 	&�
9�$��%)�%,�RD<��D<�LF:�PE�,4�0f*�R%*�"�"&�G
ryrLc�.��t���fd��}|S)z�Function decorator to capture and log any exceptions.

  This is extra insurance that analytics collection will not hinder the command
  being run upon an error.

  Args:
    func: The function to wrap.

  Returns:
    The wrapped function.
  c���	�|i|��S#t$r<}tjd�}|jd�j|�Yd}~yd}~wwxYw)N�metricsz6Exception captured in %s during metrics collection: %s)�	Exceptionr\r]r�r.)�args�kwds�er^�funcs    �rw�Wrapperz'CaptureAndLogException.<locals>.Wrapper?sU���%�
�4�
 �4�
 � ���%�� � ��+�f��l�l�K��=�=�!�%�%��%�s��	A�2A�Ar�r�r�s` rw�CaptureAndLogExceptionr�2s"���	��;�%��%�
�.ryc�.��t���fd��}|S)a\Function decorator to ignore an exception on collecting thread stats.

  An exception can happen if the thread_stats dictionary's manager gets shutdown
  before the thread's process is successfully killed. See
  _ThreadStat.AggregateStat for how we handle that case.

  Args:
    func: The function to wrap.

  Returns:
    The wrapped function.
  c�$��	�|i|��S#YyxYwr}r�)r�r�r�s  �rwr�z+CaptureThreadStatException.<locals>.WrapperYs"����
�4�
 �4�
 � ���
�s��rr�s` rw�CaptureThreadStatExceptionr�Ks"���	��;����
�.ryc�R�tj�}|r|j�yy)z9Reports the metrics that were collected upon termination.N)rLr�r�)�	collectors rw�Shutdownr�fs&���+�+�-�)��
����ryc
�<�tj�}|sy|r+|jd�s|jtd|i�|re|jd�sTdjt
|D�cgc]}|djd���c}��}|jtd|i�|jd�}|sy|r;dj|dj|��}|jtd|i�|re|jd	�sTdjt
|D�cgc]}|djd���c}��}	|jtd	|	i�|r-|jd
�s|jtd
|i�yyycc}wcc}w)a�Logs info about the gsutil command being run.

  This only updates the collector's ga_params. The actual command metric will
  be collected once ReportMetrics() is called at shutdown.

  Args:
    command_name: str, The official command name (e.g. version instead of ver).
    subcommands: A list of subcommands as strings already validated by
      RunCommand. We do not log subcommands for the help or test commands.
    global_opts: A list of string tuples already parsed by __main__.
    sub_opts: A list of command-level options as string tuples already parsed
      by RunCommand.
    command_alias: str, The supported alias that the user inputed.
  Nrrr�r�-z{0} {1}� rr#)	rLr�r�r�rir�r��striprk)
rK�subcommands�global_opts�sub_opts�
command_aliasr��opt�global_opts_string�full_command_name�sub_opts_strings
          rw�LogCommandParamsr�os���(�+�+�-�)�	�
��)�.�.�~�>�
���m�N�;�\�J�K���-�-�.>�?�����[�9�[�c��A����S�!�[�9�:�<��
���	�'�	(�*<�=�?�
�%�%�n�5�,�	�
��!�(�(��s�x�x��7L�M��
���m�N�;�=N�O�P�
�i�*�*�+B�C��h�h�v�H�&M�H�S�s�1�v�|�|�C�'8�H�&M�N�O�O�
���	�.�	/��A�C��9�/�/��@�
���m�O�<�m�L�M�A�]��!:��'Ns�,F�$Fc��tj�}|rG|j|jxxdz
cc<|jr
td��ytd��yy)z�Logs that a retryable error was caught for a gsutil command.

  Args:
    message: The RetryableErrorMessage posted to the global status queue.
  rP)r
)rN)rLr�rprL�is_service_error�LogPerformanceSummaryParams)�messager�s  rw�LogRetryableErrorr��sO���+�+�-�)��
���w�1�1�2�a�7�2����!�q�A�!�q�A�ryc��tj�}|r/|jtd|jj
i�yy)z}Logs that a fatal error was caught for a gsutil command.

  Args:
    exception: The exception that the command failed on.
  r%N)rLr�r�ri�	__class__r.)�	exceptionr�s  rw�
LogFatalErrorr��sB���+�+�-�)��
���	�}�	%�y�':�':�'C�'C�D�F�ryc�T�tj�}|r|j|�yy)a�Logs parameters necessary for a PerformanceSummary.

  gsutil periodically monitors its own threads; at the end of the execution of
  each cp/rsync command, it will present a performance summary of the command
  run.

  Args:
    **params_to_update: A dictionary. See UpdatePerformanceSummaryParams for
        details. The argument ambiguity at this level allows for flexibility in
        dealing with arguments that are processed similarly.
  N)rLr�r=)�params_to_updater�s  rwr�r��s)���+�+�-�)��
�,�,�-=�>�ryc�P�tjjdd�}tjjt�s�|s�tj�s�tdtjd�zdz�}t}|j�ddk(rtj�j }tj"tjj%t��t'td�5}|j)|�d	d	d	�y	y	y	y	#1swYy	xYw)
z�Asks a user to opt-in to data collection if a UUID file does not exist.

  If the user agrees, generates a UUID file. Will not prompt if part of SDK.
  rNr��
z�gsutil developers rely on user feedback to make improvements to the tool. Would you like to send anonymous usage statistics to help improve gsutil? [y/N]r�r�y�wN)r`rar~r�r�r�r�rr�r
�textwrap�fillr�r��uuid�uuid4�hex�CreateDirIfNeeded�dirnamer��write)�disable_prompt�enable_analytics�
text_to_writer�s    rw�'CheckAndMaybePromptForAnalyticsEnablingr��s����;�;�(�(��3M�N�.�
�'�'�.�.��
)�.�
�
(�
(�
*��T�H�M�M�	 �%!�!�$'�'�(��
#�M������"�c�)��j�j�l�&�&�m��!�!�"�'�'�/�/�/�"B�C�	
�o�s�	#�q��g�g�m��
$�	#�+�3A�
)�
$�	#�s�>D�D%c�J�|�tj�}t|dz�S)Ni�)rBro)�time_in_secs rwrYrY�s#�����)�)�+�K�	�[�4�
�	 � ry)NNNNNr})@r1�
__future__rrrr�atexit�collectionsr�	functoolsr	r\r�r{rl�rergr�r}rwr�rBr�r��	six.movesr
rr`�gslibr�gslib.metrics_tupler
�gslib.utilsr�gslib.utils.unit_utilrrr�r�rcrFrIrJra�compilerer��
expanduserr�r�r�rir2rLr�r��registerr�r�r�r�r�r�rYr�ryrw�<module>r�s{��A�&�%��'�
�#���	�
��	�
��
�����
�����&�#�5�6�9��
��"��!��*��&��+��"�r�z�z�"3�4���'�'�$�$�R�W�W�\�\�#�2J�&L�M������d���D���4���4�	�
�E���e�
��U��
�e���U���5���E���u���e������e�� ��!�"�5�#�$��+0�).�"'�$)�$*� &�!'�!'��9�
�@~

�v�~

�B�2�6���������"&�!%�!%�"�#'�	.N��.N�b�B��B�"�	F��	F��?��?�"����*!ry