All Questions
33,488 questions
0
votes
0
answers
24
views
Searching for a specific value, within a specific field, within many csv files, within many buckets, within 3 projects in GCS using Python
For example, let's say I want to search through my 3 projects > many buckets > many csv files > field that includes 'name' > for the value 'Bob'. So far I have the following code:
import ...
- 11
-3
votes
2
answers
72
views
How can I optimize a Python script to process large CSV files efficiently?
I'm working on a Python project that involves processing large CSV files (2–5 GB in size). The script reads the CSV file, performs data transformations, and writes the output to a new file. However, ...
- 11
0
votes
0
answers
46
views
Weather Data with F1 Driver Performance in Python
I am working on a data science project to analyze F1 driver performance based on weather conditions. I am using a dataset from kaggle that contains detailed information about F1 races, such as driver ...
- 133
0
votes
0
answers
52
views
How can I improve the performance of my Python script for processing large JSON files?
I am working on a Python script that processes large JSON files (~500MB) to extract specific data and save it into a CSV format. The script works, but it is slow, especially for files with nested JSON ...
- 9
0
votes
0
answers
46
views
I cannot get all data to export to CSV
# Collect batting stats for the 2022, 2023, and 2024 seasons
try:
print("Collecting batting stats from 2022 to 2024...")
batting_data = batting_stats(2021, 2024, league="all&...
1
vote
0
answers
29
views
How to merge input variables into a single list field in pydantic?
I have input data ( a csv file) that has columns that are named like item1, item2, item3
I would like to have a field in my pydantic model called 'items' that collects all the item# input variables ...
- 11
-1
votes
3
answers
73
views
How to save values to a CSV file in exponent in python [closed]
For a project in Python i have to read float values from an excel file and save it as exponential values in a CSV file by using Pandas. When i print the values, they are shown as exponent, but when i ...
- 21
1
vote
2
answers
78
views
How can I clean a year column with messy values?
I have a project I'm working on for a data analysis course, where we pick a data set and go through the steps of cleaning and exploring the data with a question to answer in mind.
I want to be able to ...
1
vote
1
answer
75
views
Problems plotting timestamps on the x-axis with Matplotlib
I am working on a Python script that loads several CSV files containing timestamps and ping data and then displays them on a plot. The X-axis is supposed to display the timestamps in HH:MM format, ...
- 340
0
votes
1
answer
39
views
Empty date error : No columns to parse from file though it looks a perfect csv file
Thank you for reading this question.
I'm a quite beginner who just started learning python.
I tried to read csv file on jupyter-notebook as following and got [No columns to parse from file] error as ...
0
votes
2
answers
82
views
How to Efficiently Optimize a Python Script for Processing Large CSV Files?
I’ve been working on a Python script to process large CSV files (500MB to 1GB). The script performs tasks like filtering rows based on column values, calculating new fields, and exporting the ...
2
votes
2
answers
74
views
How to Write a Pandas DataFrame to CSV With Strings Quoted and Integers/Empty Cells Unaltered Without Adding Escape Characters for Commas?
I am working on a Python script to write a DataFrame to a CSV file. My goal is to:
Enclose all string values in double quotes (").
Keep numeric values unchanged (no quotes).
Leave empty cells as ...
0
votes
0
answers
32
views
Apache Nifi: Unable to Merge Multiple CSVs into a Single PARQUET File using ExecuteStreamCommand Processor
I am trying to merge the multiple CSVs which are coming from upstream as a flowfiles of similar kind (same Schema) into a one PARQUET file format. Below is the Flow of my processor group. Where in ...
- 401
0
votes
1
answer
69
views
Reading an irregular column CSV file using Pandas causes errors
I have a csv file,
"==CNAME=="
""
"Tool Name","v2.1"
"Name:","MATT B"
"E-Mail:","[email protected]"
"Phone Number:&...
-1
votes
0
answers
40
views
Better way to run this python (matplotlib) code for plotting and saving images
I have the following code that reads data from a csv, plots it and then saves the plot as images. The csv file has x and y values in columns index 0 and 1 and also has RGB values for color in column ...
- 919
0
votes
1
answer
63
views
add data to specific row and colum in csv python
I have this code which I want to add data from the user input to each column and rows:
import csv
import datetime
import calendar
days = [i for i in range(1,32,1)]
days = (','.join(map(str,days)))....
- 45
1
vote
1
answer
51
views
How to prevent Pandas to_csv double quoting empty fields in output csv
I currently have a sample python script that reads a csv with double quotes as a text qualifier and removes ascii characters and line feeds in the fields via a dataframe. It then outputs the dataframe ...
- 301
1
vote
3
answers
78
views
Randomly shard large CSV into smaller CSVs
I am aiming to shuffle a csv file too large to load into the RAM of my device at once. My plan was to use csv.DictReader to go through the entire file and write each line randomly into one of a set of ...
- 13
0
votes
1
answer
38
views
Dataframe of dataframes: writing and reading
I have a set of images. In each image, a program finds objects with attributes X and type. The number of objects vary from image to image. Hence for one image I have a df_objects with N_objects rows ...
-1
votes
1
answer
35
views
I can't recognise the error in my python script for posting a csv into a MySQL DataBase
I'm attempting to send a CSV file with a large amount of columns to a MySQL database. Upon running my code I am met with this error message:
Traceback (most recent call last):
File "/home/dan-...
0
votes
2
answers
79
views
Open CSV files in subsections using loop
Edit!
I am trying to create a loop to open CSV files.
I started of by creating multiple lists of files that I want to have opened in separate dataframes:
file_list_a = [f for f in os.listdir(...
- 3
0
votes
1
answer
62
views
How to read from csv and generate excel file with same name python
Currently i am reading sql queries from column in csv file, executing them in Snowflake and then generating sql query result in separate excel file for every query. The code is working fine.
The excel ...
- 3,859
1
vote
2
answers
93
views
How to parse CSV with incorrectly separated rows to pandas dataframe
How to store properly in a pandas Datframe?
link to dataset:"https://lib.stat.cmu.edu/datasets/boston"
Snippet from data
[ 21 lines of header text]
0.00632 18.00 2.310 0 0.5380 6....
- 13
0
votes
1
answer
57
views
How to process multiple csv with pyspark in aws glue?
I'm new to pyspark and aws glue. I wrote small script based on the examples i saw to read a csv file as a dynamic pyspark frame. I would like to know, how can i read multiple csv files or all csv ...
-3
votes
2
answers
73
views
CSV to JSON using python, only selected columns, and creating new entry for each cell having multiple values from a single CSV cell?
I am very new to python and I am trying to get the following JSON output from a csv file and the csv file looks like this
date customer Ex employer emailID CIN CIN2 ...
- 13
0
votes
0
answers
62
views
Error reading csv.xz file using polars in python
I have a large file 'myfile.csv.xz' that I want to read using the polars package. When I do pl.read_csv(myfile.csv.xz) it fails with error ComputeError: invalid utf-8 sequance. If I do pl.read_csv(...
0
votes
2
answers
56
views
How to approach: csv objects specified on command line or in a file
I would like to accept 2d points provided on the command line or read from a file. Example command line positional parameter input:
( 4,-1),(0, 0) , (3,3)
example CSV file
4, -1
...
- 9,034
1
vote
0
answers
23
views
How to Correctly Categorize and Calculate Ratios for Continuous and Categorical Data in Python Without Pandas?
I'm working on a cardiovascular disease dataset analysis in Python, but I cannot use Pandas. The goal is to categorize attributes like height, weight, systolic blood pressure (ap_hi), and diastolic ...
- 11
-1
votes
2
answers
107
views
How to Optimize Large CSV Processing?
TL;DR: I’m processing a large CSV (4M rows, 510 columns) but only need a few thousand rows with all columns at the end. The whole process takes 70+ minutes and uses a lot of memory. Looking for ways ...
- 31
0
votes
0
answers
61
views
Error tokenizing data when merging multiple csv into one excel file
I'm having an issue regarding on Error tokenizing data.
I'm trying to merge multiple csv files into one excel file and my files have some special characters in it with big data.
This is the error I ...
0
votes
1
answer
59
views
Pandas dataframe is mangled when writing to csv
I have written a pipeline to send queries to uniprot, but am having a strange issue with one of the queries. I've put this into a small test case below.
I am getting the expected dataframe (df) ...
- 706
0
votes
1
answer
42
views
How to find dropdown name in HTML code for Selenium's Select class?
I want to download the CSV data shown at the end of the page using Python:
https://www.cboe.com/delayed_quotes/spx/quote_table/
Specifically, before downloading the data, I need Python to select the ...
- 1,000
-1
votes
3
answers
117
views
How to iteratively run a Python script on the contents of a CSV file
I have a Python program that takes a url as a runtime argument. I want to modify this by creating a 'wrapper' to process a CSV file containing a list of urls as the input instead of a single url. The ...
- 281
1
vote
2
answers
51
views
The generated table in the code doesnt read the csv file causing the table to show no data
The code generated a table but the table shows no value instead of reading it from the csv file.
import csv
# Load the dataset
file_path = '/Users/brosis/Downloads/AvgHouseholdElectricity.csv'
# ...
- 11
0
votes
0
answers
41
views
Handling Special Characters and Escape Sequences in CSV with Spark and PostgreSQL COPY Command
I am using PySpark to repartition a DataFrame and write it to a CSV file, which I later load into PostgreSQL using the COPY command. However, I am encountering the following error when executing the ...
0
votes
2
answers
67
views
Parsing non standard csv file in python
I am trying to parse csv file that I don't control. The file contains quoted text like ...,text ""text"",... and I want the parsed field to look like text "text".
Java ...
- 11
0
votes
0
answers
71
views
Price fetching from csv in python
I have an input csv file (top_10_stocks.csv) which has list of top 10 stocks for every month since October 2016 based on a strategy i have developed. The idea is to generate the list of these top 10 ...
- 15
1
vote
1
answer
56
views
Python CSV Reader & Writer to respect the double quotes
I'm new to Python and working on below use case:
I want my Python script to enhance a csv file.
My input.csv looks like below:
test_case,test_desc,"test data 1","test data 2"
,0,&...
- 708
0
votes
1
answer
109
views
Found character '-' instead of field delimiter ','
I'm pulling data from Postgresql into Snowflake. One of the fields contains gift messages customers can put in. One of the entries are giving me the error in the title. The gift message is the ...
- 5
1
vote
0
answers
120
views
Python polars write / read csv handling of linebreaks (eol)
I want to read in mockup data that contains a linebreak (eol) char.
Here I utilize the faker package to simulate some data.
I initialize a polars.DataFrame and write it to .csv. When I later on try to ...
- 439
-1
votes
1
answer
62
views
Using loops to extract data in Python from .csv files
I am still very new to Python. I have extracted some data - tyre sales from a
.csv file in 2023/24 (all numbers are made up). Here, the tyre sales have been summarised by month.
import pandas as pd
...
1
vote
1
answer
51
views
Why does exit() in a "with open..." block cause 31 to be printed on Terminal?
I'm trying to track down an error about exiting while a file is open. In doing so, I interactively (not in an IDE, just into Terminal after running Python) entered the following code snippet to ...
-1
votes
2
answers
111
views
Read excel with same column names with pandas
I am trying to convert an excel to csv. The excel has the following headers -
DATE,FIELD1,FEEDER BRANCH,50,100,200,500,1000,2000,FIELD2,50,100,200,500,1000,2000,FIELD3,50,100,200,500,1000,2000
As ...
- 1,957
0
votes
0
answers
45
views
How to handle cell with comma when using pd.readcsv? Error tokenizing data. C error: Expected 1 fields in line 88, saw 2
I'm reading a set of .csv files and adding them to one giant data frame called 'df', but I kept getting this error in some of my files: Error tokenizing data. C error: Expected 1 fields in line 88, ...
2
votes
0
answers
52
views
How do I put long strings into the Blob column using DBeaver's import csv
I'm trying to load the csv file to oracleDB using DBeaver's "import csv" function.
The data you want to put in is a very long string type csv cell, and the type of OracleDB is a BLOB type.
...
- 21
1
vote
1
answer
60
views
How do I count blank an filled cells in each column in a .csv file?
What I want is to count the filled and blank cells in each column of a .csv file.
This is my code:
import pandas as pd
file_path = r"C:\Users\andre\OneDrive\Documentos\Farmácia\Python\...
- 21
0
votes
0
answers
15
views
Trouble with retrieving the right HTML data onto my chart
My chart is showing completely wrong things, and i want to know where in my code that I can change what i want to retrieve, in order for it to display the correct stuff.
Is the extraction of data off? ...
- 11
1
vote
1
answer
62
views
How to prevent scientific notation when exporting a pandas dataframe to csv?
I read and modify excel files with pandas and export them as csv at the end.
One column can contain all kinds of text and sometimes a numeric string like "0123456789", note the leading zero. ...
- 317
1
vote
1
answer
35
views
Irregular column count when parsing pandas.read_clipboard()
I'm creating a script to receive plain text and turn it into csv, to later import into Anki. I'm experimenting with pandas.read_clipboard(), to try skip using txt files.
If the first line of my data ...
- 13
-2
votes
2
answers
63
views
How to merge multiple CSV/Excel files into a single file with Python inserting the first column 1 as the CSV file name?
Trying to merge >100 CSV files into a singular Excel file with the following conditions:
Specific rows of the CSV files. (The data contained is from rows 1-250)
first column of the dataframe Excel ...
