All Questions
3,214 questions
0
votes
1
answer
36
views
Indices mismatch during merge in pandas
I am trying to merge two dataframes in Python, pandas, df1 and df2.
I am trying to merge them on Column1, and then assign value of Column2 from df2 to df1.
This is my code:
df1 = df1.reset_index()
...
- 13
0
votes
0
answers
9
views
R: sp::merge does not merge sp-object and df as wanted
I am trying to join a sf-object and a df by a common column name, and have tried sp::merge without succeeding.
Sorry for formatting in advance.
I would like to keep all data from the sf object, ...
- 1
0
votes
0
answers
18
views
Merge two dataframes by replacing nans in r [duplicate]
I have the following two dataframes:
df_1
TE TS LULC DD RI Soil FA
TE NA 0.049 0.077 -0.119 -0.692 -0.035 -0.154
TS NA NA -0.559 -0.462 0.035 0.189 -0.196
LULC NA ...
2
votes
2
answers
69
views
Merge two dataframes and keep non-missing entries
I have two dataframes like this:
set.seed(1)
df1 <- data.frame(id= 1:4, sex= c("m", "m", NA, NA), somevar= letters[1:4], whocares_var= rnorm(4))
df2 <- data.frame(id= 1:6, ...
- 1,273
2
votes
2
answers
46
views
Create a new variable based on non-existing values on another variable in R dataframe
I know the title sounds very basic but I didn't find a solution by searching.
I have a dataframe like this where id refers to a participant:
id condition
1 0
1 0
1 1
1 2
2 0
2 0
2 1
3 ...
- 195
0
votes
1
answer
24
views
The syntax of how updating more than one specified column in Python Panda dataframes during a merge whenmatched condition
I am building a StreamLit appication in Snowflake. I am using a grid editor initially loading data from a table. Users can change the data in certain columns. When the users hit a submit button I want ...
- 289
3
votes
4
answers
73
views
From two dataframes with related data, fill columns in the first one (occurrences of a value in 2nd) and merge info
I have two dataframes. The first (df1) has names and the second (df2) has names and the car these persons own.
First, I want to fill the first dataframe with a new column with the occurrences of the ...
1
vote
1
answer
61
views
Pandas: Merge by common column retaining all other columns and sort by common column [duplicate]
I have two DataFrames, both of which have an x column and a y column. The x values are unique within each DataFrame, but there is some overlap between the two.
I want to merge the two DataFrames, ...
- 142
1
vote
2
answers
82
views
dataframe easily implement merge_asof
I have two dataframes:
time_start = datetime.datetime.strptime('2024-02-01 10:00:00', "%Y-%m-%d %H:%M:%S")
interval_l = [1, 7, 14, 17, 21, 22, 31]
df_l = pd.DataFrame(index = [time_start + ...
- 1,136
0
votes
0
answers
9
views
Look up values from single column in pandas df1 for multiple columns in df2 [duplicate]
I have two dataframes, df1 is a list of football teams and their average stats for the season df2 is a list of matches, each of which features two of the teams from df1. I want to be able to look up ...
- 1
0
votes
2
answers
70
views
how to make pandas merge without inner data
how to make pandas merge without inner data?
input:
inner_df = pd.merge(df1, df2, on='user', how='inner')
outer_df = pd.merge(df1, df2, on='user', how='outer')
so i don't need inner data i need only ...
user17013410
0
votes
1
answer
33
views
Is there a smart way to merge 2 DataFrames based on substring criteria?
I'm currently working with some phones numbers DataFrames and I need to merge them in a substring criteria way and has to be an left merge (meaning to have all combinations in case there is more than ...
- 1
1
vote
2
answers
60
views
how to merge dfs without duplicate lines
My dataframes are similar to below
df1:
user name length job
0 5234 Ed 7 manager
1 4793 Mark 14 admin
2 9321 Nick 8 admin
3 8149 Duncan 3 admin
4 ...
- 209
-2
votes
1
answer
71
views
Combining data with multiple dates into one row for each case
I have a bunch of data and each row has a 4 date fields. The first date field is a duplicate for the ID number and sometimes not. It looks a little like this:
ID,LName,FName,DateIn,DateOut,Days,...
0
votes
0
answers
16
views
Merging two pandas dataframe with repeated primary keys [duplicate]
I have two pandas dataframe that looks like
df1=
Class_ID Student_ID feature1
1 4 54
1 7 24
1 9 865
2 23 342
...
- 419
2
votes
1
answer
526
views
Polars Dataframe full-join (outer) on multiple columns without suffix
I have this code:
import polars as pl
df1 = pl.DataFrame({
'type': ['A', 'O', 'B', 'O'],
'origin': ['EU', 'US', 'US', 'EU'],
'qty1': [343,11,22,-5]
})
df2 = pl.DataFrame({
'type':...
- 3,183
0
votes
1
answer
562
views
How to combine datasets into a unified data frame?
Second question for my final Google Data Analytics certificate project. I have bike data from 4 quarters (split into 4 data sets) that I believe I have correctly transposed to data frames. However, ...
- 11
0
votes
1
answer
76
views
long to wide transformation without creating lists and nulls
this is a followup question from my previous post.
My dataframe arranges visit for a specific PK based on dates, with a columns for:
visit number
visit date
exam done in that visit
I am trying to ...
- 1
-1
votes
1
answer
58
views
How do I perform a smear between two dataframes in python/pandas? [duplicate]
I have two dataframes and I need to perform a smear (if that is what it's generally called). Basically the first one is smaller (5 million rows) and the other is 40 million rows. I want to add the ...
- 1
0
votes
1
answer
39
views
I want to lookup values from pivot, if value does not found then use N using Pandas Dataframe [duplicate]
Input df
This is pivot I have.I want to lookup only zero value if there in above pivot and print N if value absent.
Expected df
In excel i am able to do comparison in these 2 df with sumif function
...
- 15
2
votes
2
answers
43
views
Merge Dataframe based on substring column labeld while keep the original columns label
I have a dataframe having columns with the a label pattern (name/startDateTime/endDateTime)
import pandas as pd
pd.DataFrame({
"[RATE] BOJ presser/2024-03-19T07:30:00Z/2024-03-19T10:30:00Z&...
- 223
0
votes
2
answers
138
views
fill in null values of one pandas dataframe with another dataframe [duplicate]
I would like to fill in null values of one pandas dataframe with another dataframe (and multiple times, with multiple dataframes).
Example:
df_A
A B C
index
0 3 5....
- 53
0
votes
1
answer
101
views
When I merge 2 dataframes in R, my plots show NA cells exist but I can't find them in my Seurat object
I have a combined_seurat_object (called CJ) to which I would like to add additional columns (conditions, gene names, etc.) which is called sample_data. I am using the merge() function. Both my ...
- 3
1
vote
1
answer
143
views
Join 2 dataframes with same column but different values (Python) [duplicate]
I want to join 2 tables, with same column names, table 1 has data universe with some NA's, table 2 has valid data for which table 1 has NA values.
I want to populate NA value in table 1 based on table ...
0
votes
1
answer
31
views
Merging two dataframes of different lenght by two columns that are in both dataframes, but exist multiple times in one dataframe [closed]
I have two dataframes. One main dataframe and one supplementary dataframe.
Both of those dataframes contain the columns "year" and "density". "year" is only present as ...
1
vote
2
answers
55
views
Merge on one column if condition met else merge on another column
So consider these dfs:
data1 = [{'name': 'Muhammad', 'age_x': 20, 'city_x': 'Karachi', 'number' : 1001},
{'name': 'Ali', 'age_x': 19, 'city_x': 'Lahore', 'number' : 1002},
{'name': '...
- 175
1
vote
0
answers
69
views
How to efficiently left merge two large Dask dataframes without matching on index and while retaining partitioning in left dataframe?
I have the following two dataframes:
df_1 is a Dask dataframe containing a col_to_merge. It consists of many partitions but does not have known divisions nor an index. It is possible for me to create ...
- 11
0
votes
1
answer
37
views
Merging two Pandas dataframes without a primary keys but using latest dates instead [duplicate]
I have two pandas dataframes that looks like:
df1 records the students and their mock exam score and the mock exam date:
ID Mock_Date Student_ID Mock_score
1 14/3/...
- 419
2
votes
2
answers
65
views
join/merge multiple pandas dataframes with blending of values on one column
I have two pandas dataframes width unique column names except:
year
student_id
student_name
I would like to merge on these 3 columns (year, student_id, student_name), however sometimes the ...
- 53
0
votes
1
answer
65
views
Binding columns of two dataframes of different lengths in R tidyverse with grouped vectors [closed]
I'm trying to column bind two dataframes: the first is a time-series with numerous columns and ~10,000 rows. The second is a derivative of the first with correlation coefficients among other relevant ...
- 145
1
vote
1
answer
38
views
How to combine it into a long string based on the numerical value of the assignment in the group by, from small to large [duplicate]
I have a table, and I want to combine it into a long string based on the numerical value of the assignment in the group by, from small to large.
Here is my dataframe :
df = pd.DataFrame({'RTG_CODE':['...
0
votes
1
answer
39
views
Merging an R Data.Frame based on different columns depending on row
I have two data frames that look like:
Df1:
`
ID ID_T1 ID_T2 ID_T3 ID_T4
1 101 NA NA NA NA
2 102 11111 11111 11111 11111
3 103 22222 22222 NA NA
4 104 NA NA NA NA
5 ...
- 55
1
vote
1
answer
38
views
pandas column partial substring replace by another dataframe column if substring contains the string
I am having replace issue while I try to replace a string with a string from another dataframe.
Below is my data :
import pandas as pd
data={"ID": ["zx125", "zx137", &...
3
votes
5
answers
131
views
Join two data frames using the last row from each group
Our weather station recorded daily weather data (about 7 rows/observations) per week. We collected disease data once per week (one observation/row per week). How can I join the last row of the ...
- 323
1
vote
1
answer
36
views
after merging two dataframes, how to reassign row index?
I merged two dataframes but I want to re-sort the index so it has a sequentially larger row number by 1. So for example below, the 3162 should be 281, and so on...
278 2024-05-08 09:18:40.224469054
...
- 11
0
votes
1
answer
23
views
Horizontally merge dataframes while simultaneously making columns unique values
I have a for loop being used to fetch dataframes of data ( I provide a list of stocks and it fetches a dataframe of data for each stock. I do this by using a dictionary
d={}
fullset= pd.DataFrame ()
...
- 27
0
votes
0
answers
46
views
Joining dataframes and keeping one data column as main
I have four data frames
Df=
Roll marks grade section
> 123 45. C. A. C
> 344 67. B. A. D
> 442 90. A. B. B
> 887 80. A. B. B
Ds=
Roll ...
- 19
0
votes
0
answers
37
views
Simplifying this specific merge of dataframes?
I'm struggling to get pandas merge, join, or concat to do what I need, so I wrote a merge using for loops (I know). Sady this is a lot of data and it's simply taking way too long. How would I simplify ...
- 11
0
votes
2
answers
58
views
joining two dataframes on column value and column name
I have two dataframes that I am trying to join in some way to produce a third dataframe.
I am doing this in python.
df1:
Date
A
B
C
27/04/2023 00:00
55.6637
0.32194
0.145006
28/04/2023 00:00
57....
0
votes
1
answer
43
views
Python take data from two different dataframes, create a new dataframe and structure the data as collected from the other two
I'm very new to python and struck in an issue solving.
# First Dataframe
df1 = pd.DataFrame({
'name': ['Adam','Ashley','Adam','Don',],
'items': ['Apple','Banana','Cherry','Date'],
'...
- 13
1
vote
2
answers
39
views
Python - Edit and Replace column values in a dataframe based on a column in another dataframe
I'm new to python and struck in a problem.
df1 = pd.DataFrame({'col1': ['apple', 'banana', 'cherry', 'apple', 'cherry']})
df2 = pd.DataFrame({'col1': ['app Banana', 'Cherry', 'banana', 'apple', '...
- 13
0
votes
1
answer
60
views
how to merge dataframes in r only for rows which equal a specific value in a different column
I have 2 data frames in R.
Data frame 1 (df_1) comprises many columns, but only 3 are relevant here:
Problem |Service_Time | Imputed_Problem
----------------------------------------------
A ...
- 37
1
vote
2
answers
53
views
Merging pandas dataframes with inconsistencies
I have 2 pandas dataframes SC and SB.
SC contains data on football player physical statistics in a competition.
SB contains data on football player tracking statistics in a competition.
# Sample ...
- 251
1
vote
2
answers
132
views
Concatenate two dataframes with different headers
How can I concatenate 2 dataframes with headers of different length? I want to add the headers of the second dataframe as a row.
Below is my df1:
df1 = pd.DataFrame({'A': [1, 2], 'B': [2, 3], 'C': [3, ...
- 73
2
votes
1
answer
41
views
Append empty columns from a list to an existing dataframe
Consider this:
I have a (rather long) list of column names:
COLUMNS = [a, b, c, d]
I have a df that have some columns as in the list, and some new ones:
import pandas as pd
data = {'a': [1, 2],
...
- 175
0
votes
5
answers
94
views
Python: Merge two dataframes on certain columns using the excel equivalent of index/match
I'm trying to replicate an excel type of "index/match" in python.
I have two dfs.
df1:
RIC
Sector Code
SP500
SP400
SP600
FI
40
SP500
BRBR
30
SP400
XPEL
25
SP600
df2: (contains lookup ...
- 3
0
votes
0
answers
32
views
Is there a way to combine two data tables where the new data that needs to be appended doesn't overlap and is appended above the body of the file?
Simplifying my csv files:
Old
Timestamp,Mined,Health,Reward
2024-03-16 07:33:41,13 hours ago,100%,6.45 BTC
2024-03-15 13:13:12,1 day ago,100%,6.63 BTC
2024-03-14 12:22:30,2 days ago,100%,6.65 BTC
New
...
1
vote
1
answer
47
views
Merging two dataframes but based on string match condition
In python, I have two dataframes.
df1 looks like below:
ID Limit Comment
12 200 ['Normal']
23 202 ['Emergency']
23 203 ['Normal']
43 304 [...
- 231
0
votes
0
answers
25
views
How to replace column of one df with column of another df based on multiple matching columns? [duplicate]
I have 2 dataframes that look like this:
current_df
id email approved comment order_id country region date
0 3211 None False Others abc_123 Greece Europe ...
- 45
1
vote
2
answers
64
views
Merge two dataframes in R, utilizing one as a dictionary gives more rows
I have two dataframes in R.
One of them is huge (200k rows) and one of the columns ("fields") has a total of 160 possible values that repeat themselves.
Then I have another dataframe that ...
- 105