parquet-cli를 통해 parquet 파일의 여러가지 정보 확인 (metadata, schema 등)

parquet-tools 를 써도 되는데 parquet-cli가 좀 더 가벼워서 parquet-cli를 설치했다. (스키마만 확인하면 되는 사람...)

(env) [testuser@test-server-1 ~]$ pip install parquet-cli
Collecting parquet-cli
  Using cached parquet_cli-1.3-py2.py3-none-any.whl (3.6 kB)
Collecting pyarrow>=0.9.0.post1
  Using cached pyarrow-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25.6 MB)
Collecting pandas>=0.22.0
  Using cached pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting pytz>=2017.2
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting python-dateutil>=2.7.3
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting numpy>=1.15.4
  Using cached numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pytz, python-dateutil, numpy, pyarrow, pandas, parquet-cli
Successfully installed numpy-1.19.5 pandas-1.1.5 parquet-cli-1.3 pyarrow-6.0.1 python-dateutil-2.8.2 pytz-2023.3 six-1.16.0

 

명령어

parq filename.parquet            # view meta data
parq filename.parquet --schema   # view the schema
parq filename.parquet --head n   # view top n rows

 

참조: https://stackoverflow.com/a/66524989/8578220