All Questions
334 questions
0
votes
1
answer
46
views
How to create a mysql load file in python?
I want to use INSERT INTO statement in MySQL but my data is loaded like this (.csv):
id, name, age
1, John, 40
2, Mary, 23
3, Alex, 34
I'm using Python to prepare the load files and I'm struggling to ...
- 1
0
votes
0
answers
57
views
escaping backslashes in MySQL for CSV interpretation
In my S3, I have a CSV file that looks like this:
"id","created_at","subject","description","priority","status","recipient","...
- 3,400
0
votes
2
answers
51
views
Insert in two tables with many to many relation in a single for loop with pandas on mariaDB
I'm triyng do bulk insert data into two tables with a many to many relationship, no issue if I insert on a single table but I'm unable to insert in both at once.
CREATE TABLE IF NOT EXISTS `mydbv3`.`...
- 31
0
votes
1
answer
67
views
Escaping the result of a subquery in sql insert statements for pandas
I'm importing a csv on MariaDB using python & pandas,the isert statements need a couple of subqueries, here is the code:
empdata = pd.read_csv(
"static/files/testfile.csv",
...
- 31
-2
votes
1
answer
168
views
mysql import csv "error2 no such file or directory"
I want to import a csv file to mysql.here are my code.
I created an empty table called Sale_data and I make sure that the columns and data types are correct.
CREATE TABLE Sales_data(
Rank_number ...
- 11
0
votes
0
answers
47
views
There is no error in the code. The error is related to logins in the driver
def CSV():
import pyodbc
import pandas as pd
df = pd.read_csv("E:\\bankmgt.csv")
cnxn= pyodbc.connect(
r'DRIVER={ODBC Driver 18 for SQL Server};'
r'SERVER=server;...
0
votes
0
answers
45
views
MySql produced CSV file has all columns stacked in the first column
My MySql query is working fine and I am getting CSV output. The problem is the CSV file columns look like stacked in a single cell. I have total 24 columns. I confirmed it by opening in the Python and ...
- 4,544
0
votes
1
answer
345
views
CSV headers to MySQL columns automatically using python
I want to add CSV data to MySQL using python, but i want automatically create columns in the mysql table using columns of csv
Is there any way to create column of MySQL table with CSV using python ...
-1
votes
1
answer
62
views
Hi, I'm new to Python and a beginner all round trying to interpret this code
def list_to_csv(filename, data, header=None, skip_writing_list_header=False, dict_write_header=False, deli="\t", enc='utf-8', qc='"', method='w'):
import csv
with open(filename,...
- 9
0
votes
0
answers
115
views
Why multithreading and multiprocessing is slower than doing nothing when using LOAD DATA INFILE to mysql
I need to load a file with 2 million rows in csv format to a mysql database as fast as posible.
The process is the following:
I have a 2 million rows file.
I split it into two 1 million rows file.
...
- 79
0
votes
1
answer
1k
views
MySQL DatabaseError: 1300 (HY000): Invalid utf8mb4 character string English mixed with Chinese characters when loading file
I am facing the invalid utf8mb4 character string error when I try to do load data.
The field "descr" can contain Chinese and English characters, as well as all sorts of special characters
I ...
- 89
0
votes
1
answer
53
views
Importing CSV file to MySQL with python
I don't know what am I doing wrong here. I have already tried sevral codes but still I am not able to import the csv file to MySQL with python.I have attached the screenshot off the error message I am ...
0
votes
2
answers
123
views
How to store CSV file in database?
There is a output file from Python Pandas with a lot of columns with headers.
I need to be able handle this file by script and get CSV files in different columns positions.
For example, initial file ...
- 1
1
vote
2
answers
347
views
Move data from row 1 to row 0
I have this function written in python. I want this thing show difference between row from production column.
Here's the code
def print_df():
mycursor.execute("SELECT * FROM productions")...
- 39
0
votes
0
answers
240
views
Python: How to create & execute SQL query in Python, when query is stored in file and we need to append attribute in where clause from dict value
I have a dict which give Key (k1) and Value as (v1). And what I am looking for is to append v1 to queries stored in .sql or .csv file
def getDetailsFromRDS():
t = collect_CabItem() # stores Key ...
- 71
0
votes
2
answers
145
views
How to put date(parsed from csv) in Python to mysql?
I've made separate simple example just to check what method would work with date in my case and didn't find any solution.
we have:
db table:
CREATE TABLE `main_test` (
`id` bigint NOT NULL ...
- 1
0
votes
1
answer
1k
views
How to update values in a column MySQL database with data in a CSV file in Python?
I have a python script that stores information in a csv file. The columns are Name,Time Left, Date,ID I have a MySql database with the following columns: Name, Time Entered, Time Left, Date, ID
Time ...
0
votes
0
answers
372
views
Connection.commit() not uploading all rows to database
I am trying to populate a MySQL database table from a CSV file using PyMySQL. The CSV file has approx 948,000 rows. The script works fine but only approximately 840,000 rows appear in the database, I ...
0
votes
0
answers
777
views
Pandas .to_sql() throwing an error for duplicate entry
I am working on pandas dataframe concept to send the data on mysql server.
I have few .csv files eg. one.csv, two.csv. I am reading csv file in dataframe object and using SQLAlChemry library trying to ...
- 21
0
votes
1
answer
39
views
Missing intermediate records from the CSV file while importing it in to My SQL by using python program
I am using below python program to read the CSV file continuously and import its latest new records in to My SQL Database. The problem is missing of intermediate random records. For Example if CSV ...
0
votes
1
answer
326
views
Uploading CSV Files with spaces between words and numbers and decimals to Database
I am trying to load data (CSV Files) from S3 to MySQL RDS through Lambda. So that i have written code in Lambda so whenever csv file is upload in S3 bucket then the data will import to Database.
But ...
0
votes
1
answer
226
views
Pandas df to MySQL gives the same timestamp
I am trying to upload a CSV to MySQL by first reading the CSV using pandas's .read_csv function and then using the .to_sql function to upload it to the db table.
I have a modification_time column ...
- 3,186
0
votes
2
answers
519
views
AttributeError 'DictCursor' object has no attribute 'update'
I'm trying to import some CSV files to a table on a MySQL database. The CSV files are updated daily and my intention is to use this program in python to automate the process.
The idea is: if the ...
- 1
-1
votes
3
answers
5k
views
How to load CSV files data from S3 to MySQL RDS Using Lambda?
enter image description here
enter image description here
Let's say
Database name = EmployeeDB
In this Database we are having 5 tables which are table1, table2, table3, table4 and table5
and we also ...
0
votes
0
answers
45
views
Saving CSV to mysql
Attempting to take a CSV from a URL and store it in MYSQL. I am able to see the CSV values in terminal, but I'm hung up on loading the values into sql, currently this is only loading the first line.
...
- 1
0
votes
1
answer
906
views
Not enough parameters for the SQL statement when inserting from file to MySQL using Python
I'm trying to import from CSV files values into MySQL using Python (Jupyter Notebook from Anaconda, Python 3.8, using a dashboard_venv virtual environment with mysql.connector)
The code is here (each ...
- 63
0
votes
0
answers
49
views
Loading NULL values via MySQL 'LOAD DATA' command not working if final column in row is NULL
I have a Python process to load CSV data into MySQL. I am using the 'LOAD DATA' command, and have preprocessed my CSV file to contain '\N' instead of '' so NULL is used on the load instead of an empty ...
0
votes
2
answers
191
views
Keep getting errors while importing from csv file to mysql
import csv
import mysql.connector as mysql
marathons = []
with open ("marathon_results.csv") as file:
data = csv.reader(file)
next(data)
for rij in data:
year = rij[0],
...
-1
votes
1
answer
41
views
Python Help MySQL
I am new to Python and I am looking for a solution to populate a MySQL Database with CSV files from a Local Folder.
I would like to do a mass import to the MySQL DB
When a new CSV is added to the ...
- 23
0
votes
1
answer
866
views
How can you replace NA values with NULL while inserting data from a file to a table?
I am attempting to import data from a csv file into an existing table in my database. It seems to be working fine, except that mysql does not accept NA as a value for columns represented by an int. ...
- 15
0
votes
0
answers
40
views
How to copy csv line with extra delimiter to another file while loading to MySQL using Python
I am working on a tricky code. The client wants to load the data to MySQL but excluding the lines which have extra delimiters or have carriage return issues. I was able to complete the loading part ...
- 53
0
votes
1
answer
1k
views
Flask Upload CSV file and Insert rows into MySQL Database
I was trying to upload csv file data into mysql database. I have some error in parsecsv. Please help. I'm a complete beginner. I will really appreciate your help.
I was trying to use the code from ...
- 1
0
votes
1
answer
392
views
''Error while connecting to MySQL Not enough parameters for the SQL statement'' Upsert python mysql
I'm trying to perform Upsert query on csv dataframe in python mysql but getting error that
''Error while connecting to MySQL Not enough parameters for the SQL statement''
Here is my query:`
...
0
votes
3
answers
223
views
Reduce daily status table to only contain status changes
I have a large (570m rows) daily status table for 100k+ users. Currently it is in MySQL (or CSV). The table contains three columns: user_id, status, and date. Ideally, I'd like to reduce the table to ...
-1
votes
1
answer
643
views
1366 (HY000): Incorrect decimal value: ''8.99'' for column 'price' at row 1
create_query = '''CREATE TABLE Sales(
order_num VARCHAR(255) NOT NULL,
order_type VARCHAR(255) NOT NULL,
cust_name VARCHAR(255) NOT NULL,
cust_state VARCHAR(255) NOT NULL,
prod_category VARCHAR(255) ...
1
vote
0
answers
54
views
Error while loading data from CSV file to MySQL Database Table in Python
I've created the following table in a MySQl Databse
CREATE TABLE aux(
person VARCHAR(40) NOT NULL,
uid VARCHAR(90) NOT NULL,
CONSTRAINT cp_aux PRIMARY KEY (uid)
);
...
- 41
0
votes
2
answers
75
views
Loading CSV into dataframe results in all records becoming "NaN"
I'm new to python (and posting on SO), and I'm trying to use some code I wrote that worked in another similar context to import data from a file into a MySQL table. To do that, I need to convert it to ...
0
votes
0
answers
38
views
How can I associate panda's csv column "type" with mysql column "type" (read below)
I am creating an application where I import some csv file and "append" its data to an existing MySQL db table using pandas and sqlalchemy. Now the problem is that there could be some new ...
0
votes
0
answers
95
views
How to fix the following QuoteChar error while using Python to load data in MySQL
I was writing a code, to load data to MySQL using Python. This code helps users to directly load the data to MySQL without opening the file. As the data size are pretty big they take time to open up.
...
- 53
1
vote
1
answer
89
views
Python read mysql to csv
I would like to read a mysql database in chunks and write its contents to a bunch of csv files.
While this can be done easily with pandas using below:
df_chunks = pd.read_sql_table(table_name, con, ...
- 2,883
0
votes
3
answers
1k
views
Python JSON String to tabular formatting
The script at the bottom of the post helps me fetch and re-orient the data from JSON into a list of info like shown directly below...
I need to restructure this into tabular format for either CSV ...
2
votes
1
answer
100
views
Insert data into mysql from sql file or using csv file
I have an db.sql file and I tried inserting the data from it to my local MYSQL using the command:
mysql -u root -p chitfund < db.sql . It doesn't show any error and also doesn't print anything. But ...
- 2,084
0
votes
1
answer
246
views
dynamic insert using mysql and csv file [duplicate]
I am learning python....I am trying to load a csv file into mysql.
I am getting the below error. can someone please help?
ordinal code:
import csv
import sqlalchemy
NULL_FIELD_VALUE = r'\N'...
- 25
0
votes
1
answer
1k
views
Importing CSV with Binary UUID into MySQL Workbench [closed]
I am trying to import a CSV file with 10 million rows of UUIDs, generated in Python, into MySQL Workbench.
Firstly, I created a MySQL table with binary(16) column titled ID:
CREATE TABLE TT (
ID ...
- 912
0
votes
1
answer
236
views
Python for loop csv concurrency
I am stumbling accross large files 80.000 lines + which I do have to keep in my database. It takes like 20-30 min to push it all to my mysql database. I have a simple for loop, which just loops the ...
- 31
2
votes
1
answer
1k
views
How to write Python dictionaries to CSV when the order of keys/values varies?
How do I write a CSV file from a Python list of dictionaries, making sure the dictionary keys are always in the same order?
Specifically, I have a Python list of dictionaries (coming from a JSON API), ...
- 1,079
0
votes
2
answers
525
views
How to join two dataframe of different database(mysql and postgres) and get an output as a csv file using python?
As mentioned in the question I have the task of joining two data frames of different databases locally(MySQL and PostgreSQL) and need to get an output(dataset) in a CSV file. Let me tell you what I ...
- 1
0
votes
1
answer
140
views
How to replace special characters from CSV headers before loading to MySQL using Python?
I usually work with large datasets and am trying to code a script which will load the dataset without much manual work (I work for non-tech company). The script will run another test while loading the ...
0
votes
3
answers
89
views
How to insert data from CSV to MySQL without using parameterized queries?
I work with huge CSV data files and am planning to do few checks before inserting the data line by line to MySQL using Python. As the data files are pretty large opening the files take a hell lot of ...
0
votes
1
answer
510
views
import data repeatedly into Django models using python and SQL
I researched about importing csv data into Django models and there was no mention of SQL. I just want to check if this method has any problem.
I have many csv files which need to be uploaded into ...
- 49