434 questions
0
votes
0
answers
44
views
How to Prepare Inflow/Outflow Data as Tensors for PyTorch Dataset and Dataloader?
I’m working with a bike-sharing dataset that tracks trips between stations and I'm wondering if there's a better way to feed the (time series) data into a neural network using PyTorch.
Each record ...
- 13
1
vote
0
answers
34
views
How to Build a More Efficient DataLoader to Load Large Image Datasets?
I am trying to train a deep learning model on a very large image dataset. The model input requires a pair of images (A and B). Because my image sizes are quite large, I have resized each of them to a ...
- 189
0
votes
1
answer
38
views
Why is my DataLoader process using up to 2.6GB of virtual memory, and is there any way to reduce it?
Why is my DataLoader process using up to 2.6GB of virtual memory, and is there any way to reduce it?
Each DataLoader process takes up 2.6GB of virtual memory, and 4 processes take up 10.4GB.
from ...
- 168
1
vote
0
answers
8
views
Asynchronous parallel data loading with torch in R
I want train cnns on a big dataset via transfer learning using torch in R. Since my dataset is to big to be loaded all at once, I have to load each sample from the SSD in the dataloader. But loading ...
- 11
0
votes
0
answers
43
views
Slow Data Loading and Low GPU Utilization in PyTorch Federated Learning with Frequent Client Switching
I'm working on a federated learning project using PyTorch, focusing on medical imaging (MRI) data. Despite using an SSD, the dataset loading phase is unusually slow, and the GPU utilization remains ...
- 1
0
votes
0
answers
59
views
Pytorch Custom Dataset CPU OOM Issue
I'm having a very persistem memory issue in my dataloader than fills up memory after a arbitrary number of epochs (5-6) depending on num_workers.
I'm 85% confident that the issue is with the dataset ...
0
votes
2
answers
159
views
TypeError: 'DataLoader' object is not subscriptable in SuperGradients Trainer
I've created DataLoader objects for my training and validation datasets, but when I try to pass them to the trainer.train() method, I get the following error:
Log summary:
TypeError: 'DataLoader' ...
- 721
1
vote
2
answers
89
views
Setting random seed in Torch dataloader
I'm trying to get the torch dataloader to load the data under a specific sequence determined by the random seed 1. Here's my code:
import random
import torch.utils.data.dataset as Dataset
import torch....
- 97
0
votes
0
answers
39
views
How to convert 3D image tensor into PyTorch MNIST format for inference?
I have trained the following model with the Pytorch MNIST dataset and saved it in the end.
from torchvision import datasets
from torchvision.transforms import ToTensor
from torch.utils.data import ...
- 13
-1
votes
0
answers
33
views
Custom Data Loader For Semantic Segmentation
I was working on a project in which I wanted to load a semantic segmentation dataset I couldn't find any data loader function in tensorflow. If someone knows one kindly let me know. My directory ...
0
votes
0
answers
24
views
GraphQL filtering on nested object retrieved from database (using Dataloaders?)
I have an SQL Database that I access via EntityFramework (C#). I want to expose the objects via a GraphQL API using the Hot Choclate framework. They are to be exposed pretty much without any changes.
...
0
votes
1
answer
260
views
How to run torch dataloader in a sub-process of multiprocessing.Pool?
I want to inference model in multiprocessing, instead of use torch.distributed, how can I use multiprocessing.Pool?
I have to use num_workers=0 in subprocess to avoid error like "daemonic ...
- 31
1
vote
1
answer
53
views
How to use tf.data.interleave() with tf.py_function
I am trying to build TF data pipeline with tf.data API. I have ~100k of npz files to load and each npz has key of ["input"] and ["output"]. Some preprocessing is needed before ...
- 11
0
votes
0
answers
99
views
Uploading base64 encoded documents in ContentVersion via SOAP API
We have an export of the ContentVersion object, containing the VersionData base64 encoded file. I have rebuilt the ContentVersion object to only contain the fields: Title, Description, PathOnClient,...
- 11
0
votes
1
answer
181
views
Is there a good way to BatchMapping or use Data Loaders in Spring GraphQL utilising non-exposed fields?
Consider the below minimal GQL schema:
type query {
appointments: [Appointment!]!
}
type Appointment {
id: ID!
job: Job!
}
type Company {
id: ID!
job: Job!
}
type Job {
id: ...
- 123
1
vote
0
answers
147
views
How to pass a pytorch DataLoader to huggingface Trainer? Is that even possble?
The usual steps to use the Trainer from huggingface requires that:
Load the data
Tokenize the data
Pass tokenized data to Trainer
MWE:
data = generate_random_data(10000) # Generate 10,000 samples
...
- 381
0
votes
0
answers
17
views
Pytorch for data in dataloader can't be excuted
I write a function to get the Validation accuracy
def evaluation(loader, model, device):
model.eval()
model.to(device)
correct = 0
total = len(loader.dataset)
for data in loader:
...
0
votes
0
answers
49
views
Federated dataloader deprecated?
In the federated Learning code below, I'm using Pysyft. the goal is to distribute the FashionMNIST dataset to different clients
federated_train_loader = syft.FederatedDataLoader(
datasets....
- 63
1
vote
1
answer
297
views
PyTorch DataLoader hangs when num_workers > 0 with custom torchvision transform
I’m using PyTorch’s DataLoader to load my dataset. I’ve noticed that my program hangs indefinitely during training when I set num_workers > 0. However, it works fine when num_workers = 0.
Here’s a ...
- 11
0
votes
0
answers
84
views
PyTorch conditional paired sampling from the same dataset
I have a PyTorch dataset of (x, y) pairs where x is an input sample and y is some conditional information about x (eg. a simple supervised classification dataset where ys are labels).
I'm looking to ...
- 1
0
votes
0
answers
45
views
Trouble with passing data from DataLoader to Learner in FluxTraining.jl for UNet model
I am attempting to use FluxTraining.jl to train a UNet model u, but I am encountering difficulties in passing the data from DataLoader to the Learner correctly.
Context:
I have two datasets: one for ...
0
votes
1
answer
41
views
Issues between PyTorch DataLoader and Matplotlib's Imshow for Image Classification Task
I am currently working on a binary classification task involving image data. To begin, it is essential for me to inspect my dataset. However, I have encountered an issue with the DataLoader.
On the ...
- 31
0
votes
0
answers
134
views
Value Error when trying to make Labels to tensor
I am trying to get my custom dataloader working. I plan on using this as a yolo network. Right now, I am using transforms.v2 in pytorch for my transforms. With this, I use SanitizeBoundingBoxes() to ...
-1
votes
1
answer
137
views
Issue about PyTorch, predicting without utilizing a DataLoader return distinct predictions compared to employing a DataLoader
I try to predict a single image without using Dataloader, but I get a weird result.
This image is the result of my prediction.
With Dataloader, predicted results are consistent with labels.
However, ...
- 1
0
votes
1
answer
429
views
How to use balanced sampler for torch Dataset/Dataloader
My simplified Dataset looks like:
class MyDataset(Dataset):
def __init__(self) -> None:
super().__init__()
self.images: torch.Tensor[n, w, h, c] # n images in memmory - ...
- 1,042
1
vote
1
answer
101
views
How to keep user logged in on refresh of dashboard page using firebase onAuthStateChange in a react app and react router dom RouterProvider API
I am using react router dom RouterProvider which decouples fetching from rendering, from the official remix-run react router example of auth-router-provider, it was stated in the README.md that
we ...
0
votes
0
answers
18
views
Dataload Error - Insert failed, Can Upsert work?
I was doing an insert dataload yesterday and some of the records failed. I still need to insert about 1300 ids. Is it possible to do an upsert now for the same file? I dont want to sort the ones that ...
- 13
1
vote
1
answer
316
views
PyTorch, validation step is considerably faster if I train on the validation data, why?
I am training a FCN model, I have two dataloaders train_loader and val_loader. As you can see in the code below, I made the model train on the validation data. I did this to debug a problem I had ...
1
vote
1
answer
104
views
Validation data without targets
I have a validation dataset of images to be classified by my CNN model. I want to load these images using pytorch. torchvision.datasets.ImageFolder() function doesn't work, since there are no targets, ...
- 15
1
vote
0
answers
99
views
How can I resolve this problem with dataloaders?
I'm building some dataloaders for training and testing a machine learning model.
I have a list of tuples named "array" like this:
(Data(x=[468, 2], edge_index=[2, 1322], y=0, edge_weight=[...
0
votes
0
answers
53
views
stack expects each tensor to be equal size, but got [3, 128, 128] at entry 0 and [4, 128, 128] at entry 10
I created a custom ImageFolder with torch.utils.data.Dataset class and then converted it to a dataloader, but when I want to see one of the elements of data loader with img_custom, label_custom = next(...
- 55
0
votes
0
answers
28
views
Parallel load timed out after xx ms, stopping the load threads
We work with a PostgreSQL database (source database) housing large tables, some with multiple millions of records, used for populating a Jasper database (also PostgreSQL). The Jasper server ...
- 51
1
vote
1
answer
56
views
pandas.DataFrame.to_sql intermittently loading data partially to snowflake/database
Intermittently it happens that pandas.DataFrame.to_sql partially loads data into snowflake.
Example: DF has 25000 rows, buy the function loads only 15000 to snowflake.
Has anyone faced this issue and ...
- 11
0
votes
1
answer
259
views
Pytorch dataset - len(train_dataset) returns zero
I am trying to create a custom dataset and dataloader in pytorch, to finetune a DONUT model. For context, my dataset is organised as follows:
dataset/
├── train/
│ ├── image1.jpg
│ ├── image2.jpg
│...
1
vote
0
answers
30
views
TF keras.utils.Sequence first batch called twice
While working on a data loader for a Keras deep learning model, I added some print statements in the get_item method of the data loader. This method is in charge of returning the n-th batch to the ...
- 11
0
votes
0
answers
594
views
How to retrieve size of current batch in DataLoader?
If I am using a dataloader in Pytorch and want to define something that needs the size of the current batch, how do I access it?
The issue I have with using my defined batch size(say, r) is suppose ...
- 91
0
votes
1
answer
117
views
PermissionError Access denied
while loading some data from a network drive a permission error occurs from time to time and the script terminates with a permission error.
the error occurs in this line :
try:
data = self....
- 45
0
votes
1
answer
138
views
Modify PyTorch DataLoader to not mix files from different directories in batch
I want to load image sequences of a fixed length into batches of the same size (for example sequence length = batch size = 7).
There are multiple directories each with images from a sequence with ...
0
votes
1
answer
184
views
The Pytorch lightning finds no tuner in lr_find_results=trainer.tuner.lr_find
I'm working on using PyTorch Lightning to train a neural network with a DataLoader. I have installed PyTorch and PyTorch Lightning successfully. However, I am encountering an issue with the learning ...
- 1
0
votes
1
answer
177
views
Implementing Dynamic Data Sampling for BERT Language Model Training with PyTorch DataLoader
I'm currently in the process of building a BERT language model from scratch for educational purposes. While constructing the model itself was a smooth journey, I encountered challenges in creating the ...
0
votes
0
answers
258
views
Why does async code mess up my dataloader in a graphql resolver?
I have a dataloader that I'm using to batch requests to a service to get my user's addresses.
The loader doesn't batch requests correctly when the parent resolver uses async code.
Here's a general ...
- 139
1
vote
0
answers
219
views
Pytorch 1.13 dataloader is significantly faster than Pytorch 2.0.1
I've noticed that PyTorch 2.0.1 DataLoader is significantly slower than PyTorch 1.13 DataLoader, especially when the number of workers is set to something other than 0. I've done some research and ...
- 717
0
votes
3
answers
2k
views
Salesforce Problem with Data Loader, Java version and installing the software
i'v installed the latest version of Zulu Jdk and DataLoader to get Data Loader on my computer, but after extracting the files from the compressed file of the DataLoader, and trying to run the install....
- 21
0
votes
1
answer
124
views
GraphQL Dataloader on non-id fields?
We're using NodeJS (typescript) and GraphQL for our backend.
Therefore we rely heavily on dataloaders, and we get more and more field resolvers that needs to be resolved on something other than IDs.
...
- 1,396
0
votes
2
answers
240
views
How to do counts in batch for graphql data loader?
I'm implementing a Graphql resolver for complex datatype. I try to use a data loader to avoid N+1 problem. I have a datatype with a counter. Therefore, in one graphql query I need to perform user ...
- 893
0
votes
1
answer
459
views
Dataloader/sampler/collator to create batches based on the sample contents (sequence length)
I am converting someone else's code into a neater torch-y pipeline, using datasets and dataloaders, collate functions and samplers. While I have done such work before, I am not sure how to tackle the ...
- 28.4k
0
votes
2
answers
316
views
How to use the `shard_func` in tensorflow's `tf.data.Dataset.save`
Background:
I'm working with a large dataset saved in a non-standard format. I can write a pure python data-reader, but when called from DL dataloaders, like tf.data.Dataset, it takes forever to ...
- 1,052
0
votes
0
answers
83
views
Training a neural network without collapsing
I am trying to train a pytorch neural network to map from image space to 2D. I have the condition that I only want to use the ReLU activation function, linear layers, conv2d layers, and avgpool2d ...
- 1
0
votes
2
answers
179
views
Confusion in initialising GraphQL Dataloader in context
context: ({ req }) => { if (req) { return { ip: headers.userip, headers, userLanguage, decodedToken, dataLoaders: { seoDataLoader: createSeoDataLoader() } } } }
Here I create a createSeoDataLoader ...
3
votes
1
answer
896
views
HotChocolate v.13 [UseProjections] attribute does not work with DataLoaders
I have the following GrapqhQL query:
query {
listTenants {
totalCount
items {
tenantId
name
sites {
totalCount
items {
siteId
cmxName
...
