libcircle is an API for distributing embarrassingly parallel workloads using self-stabilization. Details on the algorithms used may be found at http://dl.acm.org/citation.cfm?id=2389114.
Dependencies
Compile and install
The current build status is: 
./configure
make all check
sudo make install
To enable output from libcircle (including fatal errors), run configure with "--enable-loglevel=number" where "number" is one of the following options:
- "1" fatal errors only.
- "2" errors and lower log levels.
- "3" warnings and lower log levels.
- "4" info messages on internal operations and lower log levels.
- "5" fine grained debug messages and lower log levels.
RPM Build and Install
To build an RPM, use the following instructions:
rpmbuild -ta libcircle-<version>.tar.gz
rpm --install <the appropriate RPM files>
Developer API Information
The basic program flow when using libcircle is the following:
- Define callbacks which enqueue or dequeue strings from the queue.
- Execute the program.
#include <libcircle.h>
{
while((data_to_process = readdir(...)) != NULL)
{
handle->enqueue(data_to_process);
}
}
{
handle->dequeue(&my_data);
...
finished_work = lstat(my_data, ...);
...
store_in_database(finished_work);
}
int rank =
CIRCLE_init(&argc, argv, CIRCLE_DEFAULT_FLAGS);
__inline__ void CIRCLE_cb_create(CIRCLE_cb func)
Definition: lib.c:117
__inline__ void CIRCLE_cb_process(CIRCLE_cb func)
Definition: lib.c:154
__inline__ void CIRCLE_finalize(void)
Definition: lib.c:266
__inline__ int32_t CIRCLE_init(int argc, char *argv[], int user_options)
Definition: lib.c:50
__inline__ void CIRCLE_begin(void)
Definition: lib.c:232
Definition: libcircle.h:46
Options
The following bit flags can be OR'ed together and passed as the third parameter to CIRCLE_init or at anytime through CIRCLE_set_options before calling CIRCLE_begin:
- CIRCLE_SPLIT_RANDOM - randomly divide items among processes requesting work
- CIRCLE_SPLIT_EQUAL - equally divide items among processes requesting work
- CIRCLE_CREATE_GLOBAL - invoke create callback on all processes, instead of just the rank 0 process
- CIRCLE_TERM_TREE - use tree-based termination detection, instead of ring-based token passing
Reductions
When enabled, libcircle periodically executes a global, user-defined reduction operation based on a time specified by the user. A final reduction executes after the work loop terminates. To use the optional reduction:
- Define and register three callback functions with libcircle:
- CIRCLE_cb_reduce_init - This function is called once on each process for each reduction invocation to capture the initial contribution from that process to the reduction.
- CIRCLE_cb_reduce_op - This function is called each time libcircle needs to combine two reduction values. It defines the reduction operation.
- CIRCLE_cb_reduce_fini - This function is called once on the root process to output the final reduction result.
- Update the value of reduction variable(s) within the CIRCLE_cb_process callback as work items are dequeued and processed by libcircle.
- Specify the time period between consecutive reductions with a call to CIRCLE_set_reduce_period to enable them.
The example below shows how to use reductions to periodically print the number of items processed. Each process counts the number of items it has processed locally. The reducton computes the global sum across processes, and it prints the global sum along with the average rate.
double reduce_start;
uint64_t reduce_count;
void reduce_init(void)
{
}
void reduce_op(const void* buf1, size_t size1, const void* buf2, size_t size2)
{
uint64_t a = *(const uint64_t*) buf1;
uint64_t b = *(const uint64_t*) buf2;
uint64_t sum = a + b;
}
void reduce_fini(const void* buf, size_t size)
{
uint64_t count = *(const uint64_t*) buf;
double now = MPI_Wtime();
double rate = 0.0;
double secs = now - reduce_start;
if (secs > 0.0) {
rate = (double)count / secs;
}
printf("Processed %llu items in %f secs (%f items/sec) ...\n",
(unsigned long long)count, secs, rate);
fflush(stdout);
}
{
handle->dequeue(&my_data);
reduce_count++;
}
reduce_start = MPI_Wtime();
reduce_count = 0;
__inline__ void CIRCLE_cb_reduce_init(CIRCLE_cb_reduce_init_fn func)
Definition: lib.c:169
void CIRCLE_set_reduce_period(int secs)
Definition: lib.c:143
__inline__ void CIRCLE_reduce(const void *buf, size_t size)
Definition: lib.c:202
__inline__ void CIRCLE_cb_reduce_op(CIRCLE_cb_reduce_op_fn func)
Definition: lib.c:180
__inline__ void CIRCLE_cb_reduce_fini(CIRCLE_cb_reduce_fini_fn func)
Definition: lib.c:191