90,499 questions
0
votes
0
answers
9
views
String parsing in DataStage
I have a problem with a DataStage Parallel Job that doesn't parse strings properly.
I have a csv file, with two columns containing json-shaped strings. The job reads it with a Sequential File stage, ...
- 11
0
votes
1
answer
21
views
Snowflake COPY option ESCAPE=NONE does not work
I have many csv.gz files. There is one detail that I could not overcome, my data has single backslash inside of it, and snowflake gets that as escape character. Because of that I pass ESCAPE=NONE or ...
- 46
-2
votes
0
answers
20
views
Checking if title can be found in SQL database and overwriting it into an CSV file [closed]
with the help of AI I wrote an Code that could read txt documents and format them into an exportable csv. Now I want the program to check if the current title can be found in the my_database.db and ...
- 1
0
votes
0
answers
5
views
Not able to filter csv files with the SAF
Using an ActivityResultLauncher to gain access to a csv file with the MIME filter "text/csv" doesn't work. In the file chooser dialog box, the file I want to open is greyed and can't be ...
- 1,804
0
votes
0
answers
16
views
Searching for a specific value, within a specific field, within many csv files, within many buckets, within 3 projects in GCS using Python
For example, let's say I want to search through my 3 projects > many buckets > many csv files > field that includes 'name' > for the value 'Bob'. So far I have the following code:
import ...
- 11
-3
votes
2
answers
62
views
How can I optimize a Python script to process large CSV files efficiently?
I'm working on a Python project that involves processing large CSV files (2–5 GB in size). The script reads the CSV file, performs data transformations, and writes the output to a new file. However, ...
- 11
0
votes
1
answer
70
views
What's the proper way to use sed to replace a whole line in a CSV file without messing it up?
case $selectedField in
fullname)
sed -i '' "/$argname/ s/\([^,]*,\)[^,]*/\1$argnewdata/" "$filename"
;;
age)
sed -i '' "/$argname/ s/\([^,]*,[^,...
0
votes
3
answers
107
views
Among many subfolders, find CSVs starting with a given string and within ZIPs or not, and merge them while adding their names in a new column with R
In a folder (path = "D:/DataLogs/), I have several subfolders. Inside these subfolders, I would like to retrieve all the csv starting only with "QCLog" and merge them (rbind) into a ...
- 656
0
votes
0
answers
46
views
Weather Data with F1 Driver Performance in Python
I am working on a data science project to analyze F1 driver performance based on weather conditions. I am using a dataset from kaggle that contains detailed information about F1 races, such as driver ...
- 133
-1
votes
0
answers
23
views
Woocommerce CSV product import does not import product categories correctly [closed]
I am having an odd issue with importing products with a CSV. I have used the import features many time but this time around its screwing.
Here is my CSV import example:
"Koi, Koi > Koi 2024, ...
- 670
-1
votes
1
answer
111
views
Using awk in ksh script
I have a requirement to split a large CSV file using KSH script. I came across below link which is using AWK utility
Split CSV files into smaller files but keeping the headers?
I tried the same as ...
- 452
0
votes
0
answers
51
views
How can I improve the performance of my Python script for processing large JSON files?
I am working on a Python script that processes large JSON files (~500MB) to extract specific data and save it into a CSV format. The script works, but it is slow, especially for files with nested JSON ...
- 9
0
votes
0
answers
45
views
I cannot get all data to export to CSV
# Collect batting stats for the 2022, 2023, and 2024 seasons
try:
print("Collecting batting stats from 2022 to 2024...")
batting_data = batting_stats(2021, 2024, league="all&...
0
votes
1
answer
25
views
How to run a YML file in GitLab CI/CD with JMX and CSV files?
I’m trying to set up a GitLab CI/CD pipeline that runs a JMeter performance test using a .jmx file and a .csv file. I have a JMX file (GIS_2CSV_sample_sya.jmx) and a CSV file (...
1
vote
0
answers
28
views
How to merge input variables into a single list field in pydantic?
I have input data ( a csv file) that has columns that are named like item1, item2, item3
I would like to have a field in my pydantic model called 'items' that collects all the item# input variables ...
- 11
0
votes
1
answer
61
views
Converting part of a multiqc_data JSON to TSV with jq
I'm trying to convert a multiqc_data.json file to a tsv file. Specifically, I only need part of data from "report_general_stats_data" section. The structure is:
{
"...
- 1
0
votes
1
answer
37
views
How can I extract specific values from a .csv-File and add them into a specific cell in a pre-exisiting dataframe/tibble in R automatically?
I want to automatically extract specific values from a .csv-File, which is generated by our measuring device, into a a dataframe/tibble in R which has a pre-defined layout. The name of the measured ...
-1
votes
3
answers
73
views
How to save values to a CSV file in exponent in python [closed]
For a project in Python i have to read float values from an excel file and save it as exponential values in a CSV file by using Pandas. When i print the values, they are shown as exponent, but when i ...
- 21
1
vote
2
answers
78
views
How can I clean a year column with messy values?
I have a project I'm working on for a data analysis course, where we pick a data set and go through the steps of cleaning and exploring the data with a question to answer in mind.
I want to be able to ...
1
vote
1
answer
75
views
Problems plotting timestamps on the x-axis with Matplotlib
I am working on a Python script that loads several CSV files containing timestamps and ping data and then displays them on a plot. The X-axis is supposed to display the timestamps in HH:MM format, ...
- 340
2
votes
1
answer
46
views
Having an issue displaying a part of a string read in from a csv file
I have a table that contains 4 columns in csv format.
I read in the rows and write them to a server by converting them to json and writing it.
The rows are built of:
name
Hash (in sha256 format)
...
0
votes
0
answers
18
views
how to write it properly? FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY 'n' [duplicate]
How to write this line properly?
FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY 'n'
or set it to a variable?
select * from employee into outfile "/var/lib/mysql-files/data.csv&...
- 21
0
votes
1
answer
39
views
Empty date error : No columns to parse from file though it looks a perfect csv file
Thank you for reading this question.
I'm a quite beginner who just started learning python.
I tried to read csv file on jupyter-notebook as following and got [No columns to parse from file] error as ...
1
vote
2
answers
68
views
Reading a CSV from a Private GitHub Repo in a Deployed Shiny App
I have an R Shiny application that currently draws on a .csv file as its main data source. That file sits in the app's folder, so that when I deploy the app to shinyapps.io with RSConnect, it uploads ...
- 41
0
votes
0
answers
10
views
Hello I have a problem with my .ldf configuration for user automation in Windows Server (ADDS)
I'm a Student in IT and i've been blocked since a few hours by this problem :
I don't understand what is the problem here is the configuration of my .ldf :
dn: CN=Henri CANTONEY,OU=Users,DC=hoywo,DC=...
- 1
0
votes
2
answers
81
views
How to Efficiently Optimize a Python Script for Processing Large CSV Files?
I’ve been working on a Python script to process large CSV files (500MB to 1GB). The script performs tasks like filtering rows based on column values, calculating new fields, and exporting the ...
0
votes
1
answer
30
views
How to compare uid nfc to csv database (tag :Mifare DESFire : ISO14443-4 : Type A, Isodep)
my app launch but nothing happened when i put a nfc tag.
The goal is normally simple, put nfc tag, compare uid to a csv database.
The Mifare DESFire : ISO14443-4 : Type A, Isodep Tag contains secured ...
2
votes
2
answers
74
views
How to Write a Pandas DataFrame to CSV With Strings Quoted and Integers/Empty Cells Unaltered Without Adding Escape Characters for Commas?
I am working on a Python script to write a DataFrame to a CSV file. My goal is to:
Enclose all string values in double quotes (").
Keep numeric values unchanged (no quotes).
Leave empty cells as ...
0
votes
0
answers
29
views
How to set up regression testing in Postman to compare JSON structures while ignoring specific values?
To load iterations from a CSV file for further testing, this script should be used.
// Load CSV data into an object
let csvData = pm.iterationData.toObject();
// Remove empty or null fields from CSV ...
0
votes
0
answers
32
views
Apache Nifi: Unable to Merge Multiple CSVs into a Single PARQUET File using ExecuteStreamCommand Processor
I am trying to merge the multiple CSVs which are coming from upstream as a flowfiles of similar kind (same Schema) into a one PARQUET file format. Below is the Flow of my processor group. Where in ...
- 401
1
vote
1
answer
138
views
Polars Rust API creating a dataframe from a string variable / reading csv with options from a string
Using the Polars Rust API, is it possible to create a DataFrame directly from a CSV string / reader while specifying options such as the separator?
Currently, I'm working around this by saving the ...
- 483
0
votes
1
answer
68
views
Reading an irregular column CSV file using Pandas causes errors
I have a csv file,
"==CNAME=="
""
"Tool Name","v2.1"
"Name:","MATT B"
"E-Mail:","[email protected]"
"Phone Number:&...
-1
votes
0
answers
40
views
Better way to run this python (matplotlib) code for plotting and saving images
I have the following code that reads data from a csv, plots it and then saves the plot as images. The csv file has x and y values in columns index 0 and 1 and also has RGB values for color in column ...
- 919
1
vote
0
answers
42
views
How to Handle Empty Fields When Parsing a CSV File in C [duplicate]
I am working on a program in C to parse data from a CSV file. The structure of my CSV file includes fields that may be empty, and I need to handle these cases properly. Here's an example of a CSV row:
...
1
vote
0
answers
59
views
PHP fgetcsv on a huge csv file, error SSL: Connection reset by peer
Php fgetcsv script to update a database by loading a csv file of 115000 rows but after about 111000 rows it stops with this error
Warning: fgetcsv(): SSL: Connection reset by peer in "script ...
0
votes
1
answer
33
views
Split column after getting multiple different CSV's into PowerBI/PowerQuery
Thank you in advance for you time and effort. I'm struggling with the following.
I want to load multiple different CSV files into PowerBI using Powerquery.These are separate files and `i do not want ...
- 3
0
votes
1
answer
63
views
add data to specific row and colum in csv python
I have this code which I want to add data from the user input to each column and rows:
import csv
import datetime
import calendar
days = [i for i in range(1,32,1)]
days = (','.join(map(str,days)))....
- 45
0
votes
1
answer
17
views
my csv file isn't getting updated after hitting submit in pythonanywhere
i have deployed my flask program using pythonanywhere.com
it is an application form
the details submitted are getting updated in the csv file when i run it using idle
but it does not get updated when ...
0
votes
1
answer
39
views
Remove double quotes from COPY vertica command
I'm using Clickhouse S3 table engine to store data in csv file in S3 storage. This csv file from time to time is read by Vertica to consume data and paste into own table
DDL of clickhouse S3 table
...
- 81
1
vote
1
answer
51
views
How to prevent Pandas to_csv double quoting empty fields in output csv
I currently have a sample python script that reads a csv with double quotes as a text qualifier and removes ascii characters and line feeds in the fields via a dataframe. It then outputs the dataframe ...
- 301
0
votes
1
answer
32
views
Jmeter adding numbers to csv labels
I have a jmeter script the reads login data from a csv, there are 100 rows of data.
The scenario is a stepped scenario which steps by 10 users every 120 seconds up to a total of 100 like so
The ...
- 140
-2
votes
2
answers
44
views
How to adjust text inside a specific column of a CSV text file as shell script?
I have a CSV source file like this one.
id;kat;kat_de;kat_en;kat_sub_de;kat_sub_en;name;strasse;plz;gemeinde;landkreis;stellplaetze;emob;linien;website;phone;betreiber;oeffnungszeiten;vbn;x_wgs84;...
2
votes
2
answers
78
views
How to put data from CSV to JSON?
I have a CSV containing two "columns" 'User id' and 'email' example:
User id,email
1234-1234-1234,[email protected]
321-1235-44432,[email protected]
322136231345,[email protected]
...
- 113
0
votes
0
answers
14
views
Special Character csv in ODI12c
I need your support. I'm trying to load a csv file in ANSI format in ODI, but when it loads ODI can't read special characters like apostrophe or accents. If I try to open the file in notepad I see the ...
-1
votes
1
answer
78
views
Upload Google Sheet with App Script to FTP Server
I have a Google Sheet in my Google Drive, and I would like to know if there is a way to automatically upload this sheet to an FTP server (created with FileZilla) using a trigger.
2
votes
3
answers
114
views
Weird data format parsing
I have a 'csv' file to parse (or actually a ton of it). The format is quite unorthodox:
# comment
unquoted text;timestamp1;k=v;k2=v2;....;k5=v5
unquoted text;timestamp2;k=v1;k2=v12;....;kn=vn
Output ...
- 10.3k
-1
votes
1
answer
88
views
What to do to avoid rounding off decimal in SqlBulkCopy from C# to SQL Server?
I am importing .csv file data into a SQL Server table using SqlBulkCopy. My .csv file can be of any format as I can't determine the columns in it and the datatypes of it as same column can have ...
- 7
0
votes
2
answers
58
views
VBA in Access - Comand: docmd.TransferText - Exporting table to CSV text file - Delimiting data with ; in place of ,
good luck for all, :)
I'm trying to export an Access table to a CSV file, but the delimiter I want to use is a ; in spite of the default ,.
I have a table, tUsuarias, like this,
Well, for export that ...
0
votes
1
answer
81
views
Converting a UTF-8-CSV to UTF-16 (Unicode) in PowerShell
To whom it may concern,
I want to convert my CSV, which is in UTF-8, into UTF-16 using PowerShell. For this task, I wrote the following line of code:
$schuelerCSVUTF8 = Get-Content -Path "C:\tmp\...
1
vote
2
answers
87
views
Using variables in awk within echo statement that prints into a file
We use a script that prints bash commands into a file that is then run on an HPC system. It is supposed to run through a large text file containing geographic coordinates separated by whitespace and ...
- 353