File: //lib/google-cloud-sdk/lib/surface/storage/__pycache__/diagnose.cpython-312.pyc
�
��f< � � � d Z ddlZddlZddlmZ ddlmZ ddl m
Z
ddl m Z ddl mZ
ddl mZ ddl mZ dd lmZ dd
lmZ dd lmZ dd lmZ dd
lmZ ddlmZ ddlmZ ddl m Z dZ dej> dej@ fd�Z!d� Z" G d� dejF � Z$e jJ G d� de jL � � Z'y)z,Diagnose Google Cloud Storage common issues.� N)�errors)�client)� arg_parsers)�base)� errors_util)� storage_url)� direct_connectivity_diagnostic)� download_throughput_diagnostic)� export_util)�latency_diagnostic)� system_info)� upload_throughput_diagnostic)�gcs_resource_reference)�log�1GB�
bucket_url�returnc �� � t j � } |j | j � S # t j
$ r(}t
j d| j � �� |�d}~ww xY w)a Fetches the bucket resource for the given bucket storage URL.
Args:
bucket_url: The URL object to get the bucket resource for.
Returns:
The bucket resource for the given URL.
Raises:
FatalError: If the bucket resource could not be fetched.
z)Bucket metadata could not be fetched for N)�gcs_json_client�
JsonClient�
get_bucket� bucket_name�
api_errors�
CloudApiError�command_errors�
FatalError)r �
gcs_client�es � lib/surface/storage/diagnose.py�get_bucket_resourcer ( sk � � �)�)�+�*�
� � � ��!7�!7� 8�8�� � !� !�
�
�
#�
#�
3�J�4J�4J�3K�L� � �
��
�s �1 �A,�#A'�'A,c � � t j | j t j | j
� � | j ry| j rlt j j | j � r)t j j | j � st d| j � d�� �yyy)z@Validates and raises error if the command arguments are invalid.zInvalid destination path: z . Please provide a valid path.N)
r � raise_error_if_not_gcs_bucket� command_pathr �storage_url_from_string�url�export� destination�os�path�exists�isdir�
ValueError)�argss r �_validate_argsr. ? s� � �
�+�+�
����<�<�T�X�X�F��
�k�k�
�
�
�
�'�'�.�.��)�)�
*�r�w�w�}�}�T�=M�=M�/N� �
$�T�%5�%5�$6� 7� � � � 0O� � � c � � e Zd ZdZdZdZdZdZy)�TestTypezEEnum class for specifying performance test type for diagnostic tests.�DIRECT_CONNECTIVITY�DOWNLOAD_THROUGHPUT�UPLOAD_THROUGHPUT�LATENCYN)�__name__�
__module__� __qualname__�__doc__r2 r3 r4 r5 � r/ r r1 r1 R s � �M�-��-��)�� �'r/ r1 c �6 � e Zd ZdZddd�Zed� � Zd� Zd� Zy) �Diagnosez Diagnose Google Cloud Storage.a�
The diagnose command runs a series of diagnostic tests for common gcloud
storage issues.
The `URL` argument must name an exisiting bucket for which the user
already has write permissions. Standard billing also applies.
Several test files/objects will be uploaded and downloaded to this bucket
to gauge out the performance metrics. All the temporary files will be
deleted on successfull completion of the command.
By default, the command executes `DOWNLOAD_THROUGHPUT`,
`UPLOAD_THROUGHPUT` and `LATENCY` tests. Tests to execute can be overriden
by using the `--test-type` flag.
Each test uses the command defaults or gcloud CLI configurations for
performing the operations. This command also provides a way to override
these values via means of different flags like `--process-count`,
`--thread-count`, `--download-type`, etc.
The command outputs a diagnostic report with sytem information like free
memory, available CPU, average CPU load per test, disk counter deltas and
diagnostic information specific to individual tests on successful
completion.
a
The following command runs the default diagnostic tests on ``my-bucket''
bucket:
$ {command} gs://my-bucket
The following command runs only UPLOAD_THROUGHPUT and DOWNLOAD_THROUGHPUT
diagnostic tests:
$ {command} gs://my-bucket --test-type=UPLOAD_THROUGHPUT,DOWNLOAD_THROUGHPUT
The following command runs the diagnostic tests using ``10'' objects of
``1MiB'' size each with ``10'' threads and ``10'' processes at max:
$ {command} gs://my-bucket --no-of-objects=10 --object-size=1MiB
--process-count=10 --thread-count=10
The following command can be used to bundle and export the diagnostic
information to a user defined ``PATH'' destination:
$ {command} gs://my-bucket --export --destination=<PATH>
)� DESCRIPTION�EXAMPLESc �z � |j d� |j dt d�� |j dt j t t D �cg c] }|j �� c}� �� ddg � � |j d
t t j D �cg c] }|j �� c}� t j j d � � |j d
d�� |j dt t j D �cg c] }|j �� c}� t j j d� � |j dt j d�� d�� |j dt j d�� d�� |j dd�� }|j ddt j d�� d � � |j ddd d� � }|j d t j t �!� d"�� |j d#d$t j t j t �!� �%� d&�'� |j dd(�� }|j d)d*dd+�,� |j d-t d.�� |j" j% d/� y c c}w c c}w c c}w )0NFr% z+Bucket URL to use for the diagnostic tests.)�type�helpz --test-type)�choices�
TEST_TYPESaq
Tests to run as part of this diagnosis. Following tests are supported:
DIRECT_CONNECTIVITY: Run a test upload over the Direct Connectivity
network path and run other diagnostics if the upload fails.
DOWNLOAD_THROUGHPUT: Upload objects to the specified bucket and record
the number of bytes transferred per second.
UPLOAD_THROUGHPUT: Download objects from the specified bucket and
record the number of bytes transferred per second.
LATENCY: Write the objects, retrieve their metadata, read the objects,
and record latency of each operation.
)r@ �metavarrA �defaultz--download-typeau
Download strategy to use for the DOWNLOAD_THROUGHPUT diagnostic test.
STREAMING: Downloads the file in memory, does not use parallelism.
`--process-count` and `--thread-count` flag values will be ignored if
provided.
SLICED: Performs a [sliced download](https://cloud.google.com/storage/docs/sliced-object-downloads)
of objects to a directory.
Parallelism can be controlled via `--process-count` and `--thread-count`
flags.
FILE: Download objects as files. Parallelism can be controlled via
`--process-count` and `--thread-count` flags.
)rB rE rA z --logs-pathzNIf the diagnostic supports writing logs, write the logs to this file location.)rA z
--upload-typea�
Upload strategy to use for the _UPLOAD_THROUGHPUT_ diagnostic test.
FILE: Uploads files to a bucket. Parallelism can be controlled via
`--process-count` and `--thread-count` flags.
PARALLEL_COMPOSITE: Uploads files using a [parallel
composite strategy](https://cloud.google.com/storage/docs/parallel-composite-uploads).
Parallelism can be controlled via `--process-count` and `--thread-count`
flags.
STREAMING: Streams the data to the bucket, does not use parallelism.
`--process-count` and `--thread-count` flag values will be ignored if
provided.
z--process-count� )� lower_boundz;Number of processes at max to use for each diagnostic test.z--thread-countz9Number of threads at max to use for each diagnostic test.zObject properties:)� sort_argsrA z--object-countTz2Number of objects to use for each diagnostic test.)�requiredr@ rA zObject size properties:)�mutexrH rA rI z
--object-size)� upper_boundz,Object size to use for the diagnostic tests.z--object-sizes� OBJECT_SIZES)� element_typez�
List of object sizes to use for the tests. Sizes should be
provided for each object specified using `--object-count` flag.
)rD r@ rA zExport diagnostic bundle.z--export�
store_truea
Generate and export a diagnostic bundle. The following
information will be bundled and exported into a gzipped tarball
(.tar.gz):
- Latest gcloud CLI logs.
- Output of running the `gcloud storage diagnose` command.
- Output of running the `gcloud info --anonymize` command.
Note: This command generates a bundle containing system information like
disk counter detlas, CPU information and system configurations. Please
exercise caution while sharing.
)�actionrI rA z
--destinationzCDestination file path where the diagnostic bundle will be exported.a
table(
name,
operation_results[]:format='table[box](name,payload_description:wrap,result:wrap)'
)
)� SetSortArgs� add_argument�strr �ArgList�sortedr1 �value�"download_throughput_diagnostic_lib� DownloadType�FILE� upload_throughput_diagnostic_lib�
UploadType�
BoundedInt� add_group�
BinarySize�_OBJECT_SIZE_UPPER_BOUND� display_info� AddFormat)�cls�parser�option�object_properties_group� object_size_properties_group� export_groups r �Argsz
Diagnose.Args� s � �
���u� �
���
�
�
:� � �
����
�
�
��x�@�x�V�F�L�L�x�@�A�
� � � �+ � �. �����<�I�I�
�I��
�L�L�I�
� � 3�?�?�D�D� � � �. ����
� � � �����:�E�E�
�E��
�L�L�E�
� � 1�;�;�@�@� � � �0 ����
�
#�
#��
2�
J� � �
����
�
#�
#��
2�
H� � � %� .� .�� 2� /� �� �(�(���
�
#�
#��
2�
A� )� � $;�#D�#D���
&�� $E� $� � !�-�-��
�
#�
#�0H�
I�
;� .� �
!�-�-�� �
�
�
�$�/�/� 4��
�
� .� � �#�#�� 9� $� �L� � � ���� � � �$ � � ��
�
� � � ���!�!� #&� '��m A��,
��<
s �J.� J3� J8c �� � d}|j rl|j r:t |j � |j k7 r t d� �|j }n&|j r|j g|j z }t j � }g }t j ||� 5 t j j |v rAt j ||� }|j � |j |j � t j j |v rxt# j$ |j& � }t# j( ||||j* |j, �� } | j � |j | j � t j. j |v rxt1 j2 |j4 � }
t1 j6 ||
||j* |j, �� } | j � |j | j � |j t j8 |� � |cddd� S # 1 sw Y yxY w)z+Runs test with system performance tracking.NzCNumber of object sizes provided should match the number of objects.)�
process_count� thread_count) � object_count� object_sizes�lenr, � object_sizer
�get_system_info_provider�)get_disk_io_stats_delta_diagnostic_resultr1 r5 rU �latency_diagnostic_lib�LatencyDiagnostic�execute�append�resultr3 rV rW �
download_type� DownloadThroughputDiagnosticri rj r4 rY rZ � upload_type�UploadThroughputDiagnostic�!get_system_info_diagnostic_result) �selfr- �
url_object� tests_to_runrl �system_info_provider� test_resultsr rv r
rx r s r �$_run_tests_with_performance_trackingz-Diagnose._run_tests_with_performance_tracking: s! � � �L� ���
� � � �t� � � !�T�%6�%6� 6��� � �
�*�*�,� � � � �(�(�)�D�,=�,=�=� �&�?�?�A���L� � >� >� �l�
�
� � � � �<� /� 3� E� E� � �
�� �"�"�$���� .� 5� 5�6� � %� %� +� +�|� ;�:�G�G� � � �
�
�
/� K� K�� � � "� 0� 0� !� .� .�
� '� '�.�.�0���� :� A� A�B� � #� #� )� )�\� 9�6�A�A� � � �
� �
-� G� G��� � "� 0� 0� !� .� .�
� %� %�,�,�.���� 8� ?� ?�@� ���
�
7�
7�8L�
M�� �a
�
�
�s
�'F+I � I%c � � t j j t j j t j j g}t |� t
j |j � }t |� }t j j d|j � d�� t j j d|j � �� t j j d|j � �� |j r
|j }n|}|t j" j gk( rg }n| j% |||� }t j" j |v rLt' j( ||j* �� }|j- � |j/ |j0 � |j2 rnt j j d� t5 j6 ||j8 � }t j j dj; |� � y t j j d� |S ) NzUsing z! bucket for the diagnostic tests.zBucket location : zBucket storage class : )� logs_pathz Exporting diagnostic bundle...z-Successfully exported diagnostic bundle to {}z Generating diagnostic report...) r1 r3 rU r5 r4 r. r r$ r% r r �status�Print�name�location�default_storage_class� test_typer2 r� r � DirectConnectivityDiagnosticr� rs rt ru r& r �export_diagnostic_bundler' �format) r{ r- �
default_testsr| �bucket_resourcer} r �direct_connectivity� export_paths r �Runz Diagnose.Run� s� � � �$�$�*�*���� � ��"�"�(�(��M� �4�� �4�4�T�X�X�>�J�)�*�5�O��J�J���
� �%�%�&�&G�H�� �J�J���)�/�*B�*B�)C�D�E��J�J���
!�/�"G�"G�!H�I�� �~�~��^�^�l�"�l���4�4�:�:�;�;��l��>�>�
�
�L��l� �#�#�)�)�\�9�
(�
E�
E� � ��� � � �!�!�#����-�4�4�5� �{�{� �j�j���7�8� �8�8�
��(�(��k�
�j�j���
9�
@�
@��
M�� ��J�J���6�7� �r/ N) r6 r7 r8 r9 �
detailed_help� classmethodrg r� r� r: r/ r r< r< [ s: � �&�
�0
�30�-�d �f'� �f'�PD�L5r/ r< )(r9 �enumr( � googlecloudsdk.api_lib.storager r �'googlecloudsdk.api_lib.storage.gcs_jsonr r �googlecloudsdk.callioper r �"googlecloudsdk.command_lib.storager r r �+googlecloudsdk.command_lib.storage.diagnoser r
rV r r rq r
r rY �,googlecloudsdk.command_lib.storage.resourcesr �googlecloudsdk.corer r^ �
StorageUrl�GcsBucketResourcer r. �Enumr1 �DefaultUniverseOnly�Commandr< r: r/ r �<module>r� s� �� 3� � � ?� M� /� (� G� :� :� V� |� C� d� C� x� O� #� � �
��&�&�
��-�-�
�.�&�t�y�y� � ���Y�t�|�|� Y� �Yr/