Friday, December 4, 2009

TestPRandomGraphSource

I am playing with TestPRandomGraphSource on the cluster here. But already I am getting some issues:

$ mpirun --hostfile ~/mpicluster -x DISPLAY=localhost:10.0 /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource ~/Kitware/VTK-gcc
Testing simple random generator (100, 200)...
...done.
Testing simple tree+random generator (100, 200)...
...done.
Breadth-first search... done in 0.0102899 seconds
Vertex coloring... done in 0.00678277 seconds
Verifying vertex coloring... done in 0.00980902 seconds
Connected components...[dhcp-67-182:15734] *** Process received signal ***
[dhcp-67-181:16211] *** Process received signal ***
[dhcp-67-181:16211] Signal: Segmentation fault (11)
[dhcp-67-181:16211] Signal code: (128)
[dhcp-67-181:16211] Failing at address: (nil)
[dhcp-67-182:15734] Signal: Segmentation fault (11)
[dhcp-67-182:15734] Signal code: (128)
[dhcp-67-182:15734] Failing at address: (nil)
[dhcp-67-182:15734] [ 0] /lib/libc.so.6 [0x7fa832501fd0]
[dhcp-67-181:16211] [ 0] /lib/libc.so.6 [0x7f08aaf61fd0]
[dhcp-67-182:15734] [ 1] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost3putINS_25shared_array_property_mapINS_18default_color_typeENS_16vtkGraphIndexMapEEERS2_xS2_EEvRKNS_14put_get_helperIT0_T_EET1_RKT2_+0x2a) [0x7fa838518ce4]
[dhcp-67-181:16211] [ 1] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost3putINS_25shared_array_property_mapINS_18default_color_typeENS_16vtkGraphIndexMapEEERS2_xS2_EEvRKNS_14put_get_helperIT0_T_EET1_RKT2_+0x2a) [0x7f08b0f78ce4]
[dhcp-67-182:15734] [ 2] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost18depth_first_searchINS_14local_subgraphIKP18vtkUndirectedGraphEENS_6detail19components_recorderINS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_25shared_array_property_mapINS_18default_color_typeESG_EEEEvRKT_T0_T1_NS_12graph_traitsISN_E17vertex_descriptorE+0xd7) [0x7fa838537c3e]
[dhcp-67-181:16211] [ 2] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost18depth_first_searchINS_14local_subgraphIKP18vtkUndirectedGraphEENS_6detail19components_recorderINS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_25shared_array_property_mapINS_18default_color_typeESG_EEEEvRKT_T0_T1_NS_12graph_traitsISN_E17vertex_descriptorE+0xd7) [0x7f08b0f97c3e]
[dhcp-67-182:15734] [ 3] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost18depth_first_searchINS_14local_subgraphIKP18vtkUndirectedGraphEENS_6detail19components_recorderINS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_15graph_visitor_tENS_11no_propertyEEEvRKT_RKNS_16bgl_named_paramsIT0_T1_T2_EE+0x12b) [0x7fa838537ff0]
[dhcp-67-181:16211] [ 3] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost18depth_first_searchINS_14local_subgraphIKP18vtkUndirectedGraphEENS_6detail19components_recorderINS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_15graph_visitor_tENS_11no_propertyEEEvRKT_RKNS_16bgl_named_paramsIT0_T1_T2_EE+0x12b) [0x7f08b0f97ff0]
[dhcp-67-182:15734] [ 4] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost20connected_componentsINS_14local_subgraphIKP18vtkUndirectedGraphEENS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_15property_traitsIT0_E10value_typeERKT_SI_NS_11enable_if_cIXsrNS_19is_base_and_derivedINS_21vertex_list_graph_tagENS_12graph_traitsISL_E18traversal_categoryEEE5valueENS_5graph6detail12no_parameterEE4typeE+0x6b) [0x7fa8385380a2]
[dhcp-67-181:16211] [ 4] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost20connected_componentsINS_14local_subgraphIKP18vtkUndirectedGraphEENS_21iterator_property_mapIN9__gnu_cxx17__normal_iteratorIPmSt6vectorImSaImEEEENS_16vtkGraphIndexMapEmRmEEEENS_15property_traitsIT0_E10value_typeERKT_SI_NS_11enable_if_cIXsrNS_19is_base_and_derivedINS_21vertex_list_graph_tagENS_12graph_traitsISL_E18traversal_categoryEEE5valueENS_5graph6detail12no_parameterEE4typeE+0x6b) [0x7f08b0f980a2]
[dhcp-67-182:15734] [ 5] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost5graph11distributed9cc_detail29parallel_connected_componentsIP18vtkUndirectedGraphNS_8parallel24distributed_property_mapINS1_17mpi_process_groupENS_18vtkVertexGlobalMapEP14vtkIdTypeArrayEEEEvRT_T0_+0x188) [0x7fa83853823c]
[dhcp-67-181:16211] [ 5] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN5boost5graph11distributed9cc_detail29parallel_connected_componentsIP18vtkUndirectedGraphNS_8parallel24distributed_property_mapINS1_17mpi_process_groupENS_18vtkVertexGlobalMapEP14vtkIdTypeArrayEEEEvRT_T0_+0x188) [0x7f08b0f9823c]
[dhcp-67-182:15734] [ 6] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN26vtkPBGLConnectedComponents11RequestDataEP14vtkInformationPP20vtkInformationVectorS3_+0x537) [0x7fa838502887]
[dhcp-67-181:16211] [ 6] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkParallel.so.5.5(_ZN26vtkPBGLConnectedComponents11RequestDataEP14vtkInformationPP20vtkInformationVectorS3_+0x537) [0x7f08b0f62887]
[dhcp-67-182:15734] [ 7] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN17vtkGraphAlgorithm14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x54) [0x7fa833bc901e]
[dhcp-67-181:16211] [ 7] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN17vtkGraphAlgorithm14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x54) [0x7f08ac62901e]
[dhcp-67-182:15734] [ 8] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkExecutive13CallAlgorithmEP14vtkInformationiPP20vtkInformationVectorS3_+0x7d) [0x7fa833ba7dad]
[dhcp-67-181:16211] [ 8] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkExecutive13CallAlgorithmEP14vtkInformationiPP20vtkInformationVectorS3_+0x7d) [0x7f08ac607dad]
[dhcp-67-182:15734] [ 9] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_+0x65) [0x7fa833b91fc5]
[dhcp-67-181:16211] [ 9] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_+0x65) [0x7f08ac5f1fc5]
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 16211 on node mpi1 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[dhcp-67-182:15734] [10] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x4a4) [0x7fa833b96226]
[dhcp-67-181:16211] [10] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x4a4) [0x7f08ac5f6226]
[dhcp-67-182:15734] [11] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x34d) [0x7fa833d0cc7b]
[dhcp-67-181:16211] [11] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x34d) [0x7f08ac76cc7b]
[dhcp-67-182:15734] [12] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline10UpdateDataEi+0x2bb) [0x7fa833b95d73]
[dhcp-67-181:16211] [12] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline10UpdateDataEi+0x2bb) [0x7f08ac5f5d73]
[dhcp-67-182:15734] [13] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline6UpdateEi+0xc9) [0x7fa833d0b717]
[dhcp-67-181:16211] [13] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline6UpdateEi+0xc9) [0x7f08ac76b717]
[dhcp-67-182:15734] [14] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkExecutive6UpdateEv+0x3d) [0x7fa833ba87f7]
[dhcp-67-181:16211] [14] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkExecutive6UpdateEv+0x3d) [0x7f08ac6087f7]
[dhcp-67-182:15734] [15] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline6UpdateEv+0x15) [0x7fa833b9413f]
[dhcp-67-181:16211] [15] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN23vtkDemandDrivenPipeline6UpdateEv+0x15) [0x7f08ac5f413f]
[dhcp-67-182:15734] [16] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline6UpdateEv+0x15) [0x7fa833d03341]
[dhcp-67-181:16211] [16] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN32vtkStreamingDemandDrivenPipeline6UpdateEv+0x15) [0x7f08ac763341]
[dhcp-67-182:15734] [17] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkAlgorithm6UpdateEv+0x26) [0x7fa833b1a6aa]
[dhcp-67-181:16211] [17] /home/mpiuser/Kitware/VTK-gcc/bin/libvtkFiltering.so.5.5(_ZN12vtkAlgorithm6UpdateEv+0x26) [0x7f08ac57a6aa]
[dhcp-67-182:15734] [18] /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource(main+0x12a9) [0x4654dd]
[dhcp-67-181:16211] [18] /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource(main+0x12a9) [0x4654dd]
[dhcp-67-182:15734] [19] /lib/libc.so.6(__libc_start_main+0xfd) [0x7fa8324eeabd]
[dhcp-67-181:16211] [19] /lib/libc.so.6(__libc_start_main+0xfd) [0x7f08aaf4eabd]
[dhcp-67-182:15734] [20] /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource [0x463f39]
[dhcp-67-181:16211] [20] /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource [0x463f39]
[dhcp-67-182:15734] *** End of error message ***
[dhcp-67-181:16211] *** End of error message ***
[dhcp-67-182][[20651,1],2][../../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv] [dhcp-67-181][[20651,1],0][../../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)


sigh :(

1 comment:

Unknown said...

Trying to check valgrind output:

$ mpirun --hostfile ~/mpicluster -x DISPLAY=localhost:10.0 xterm -e "valgrind --log-file=val.log /home/mpiuser/Kitware/VTK-gcc/bin/TestPRandomGraphSource" ~/Kitware/VTK-gcc
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 15745 on
node mpi2 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------