Template Class DataLoaderBase¶
Defined in File base.h
DataLoaderBase(DataLoaderOptions options, std::unique_ptr<Dataset> main_thread_dataset = nullptr)¶
Constructs a new DataLoader from a
datasetto sample from,
optionsto configure the DataLoader with, and a
samplerthat specifies the sampling strategy.
Returns an iterator into the DataLoader.
The lifetime of the iterator is bound to the DataLoader. In C++ standards language, the category of the iterator is
OutputIterator. See https://en.cppreference.com/w/cpp/named_req/OutputIterator for what this means. In short: you may increment the iterator and dereference it, but cannot go back, or step forward more than one position at a time. When the DataLoader is exhausted, it will compare equal with the special “sentinel” iterator returned by
DataLoader::end(). Most of the time, you should only use range-for loops to loop over the DataLoader, but standard algorithms like
std::copy(dataloader.begin(), dataloader.end(), output_iterator)are supported too.
Returns a special “sentinel” iterator that compares equal with a non-sentinel iterator once the DataLoader is exhausted.
Joins the DataLoader’s worker threads and drains internal queues.
This function may only be invoked from the main thread (in which the DataLoader lives).
get_batch_request() = 0¶
Subclass hook for getting the next batch request.
The stateless case will ask the sampler for a new batch request (e.g. a vector of indices), while the stateful one will simply return the batch size.
Resets the internal state of the DataLoader, optionally pre-fetching new jobs.
requested_jobsmany new batches to be fetched.
The actual number of jobs scheduled may be less if the DataLoader exhausts.
Schedules the maximum number of jobs (based on the
Returns the next batch of data, or an empty
optionalif the DataLoader is exhausted.
This operation will block until a batch is available if one is still expected.
Convenience method that calls
shuttle_.push_job()with the next sequence number.
The dataset for the main thread, only has a value if the number of worker threads was configured as zero, meaning the main thread has to do all the work (synchronously).
NOTE: Really want this to be on the heap when empty, therefore
The sequence number for the next batch to be retrieved from the dataset.
DataShuttlewhich takes care of the life cycle of a job.
Sequencer, which handles optional ordering of batches.
True if the DataLoader has joined its worker threads.
Job: public torch::data::DataLoaderBase<Dataset, Batch, BatchRequest>::Sequenced¶
Result: public torch::data::DataLoaderBase<Dataset, Batch, BatchRequest>::Sequenced¶
The finished result of a job.
Result() = default¶
Result(std::exception_ptr exception, size_t sqn)¶
Simple mix-in to give something a sequence number.