This is the mail archive of the ecos-discuss@sources.redhat.com mailing list for the eCos project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: JFFS2 fragment corruption during gc?


Hi Mark,

I've tried your suggested solution and my test runs like a dream, well at least almost.
There is another error present that has to do with the return of ENOSPC, this increases the use count on a node so that the umount() function fails.
But that is another story for the moment.


But if I'm not mistaken the gc.c file is a general JFFS2 file, so the alterations should really be in the fs-ecos.c file instead.
That's if the Linux folks are not having the same type of problems.


//Alf Nilsson

Mark Hamilton wrote:

Alf,
 I've seen this problem as well. I sent an email to the ecos-discussion
group a few weeks back asking if my fix made sense. From what I can tell,
there is a bug with the eCos port of the garbage collection. The eCos group
directed me to the JFFS mailing list. I sent my email there but never
received a response.

Below is my original email which includes my fix. I would appreciate it if
you could tell me if this solves your problem.

I'm using the JFFS file system and I'm having problems with files being
truncated during garbage collection. I believe I've tracked down the
problem. I'm hoping someone can give me positive feedback on my fix. The
problem is in this snippet of code below:


Jffs2_gc_fetch_page reads 4K of data into a static buffer. The static buffer is hidden in the jffs2_gc_fetch_page function. The problem is when the writebuf pointer is calculated. The offset is used again to reference into the pg_ptr. You can image when start is equal to 4K that writebuf will extend beyond the end of the pg_ptr valid memory. Offset is set to start just before the while loop.

I made a comment below with what I think the fix should be.
Am I missing something?

pg_ptr = jffs2_gc_fetch_page(c, f, start, &pg);
if (IS_ERR(pg_ptr)) {
printk(KERN_WARNING "read_cache_page() returned error:
%ld\n",
PTR_ERR(pg_ptr));
return PTR_ERR(pg_ptr);
}
offset = start;
while(offset < orig_end) {
uint32_t datalen;
uint32_t cdatalen;
char comprtype = JFFS2_COMPR_NONE;
ret = jffs2_reserve_space_gc(c, sizeof(ri) +
JFFS2_MIN_DATA_LEN, &phys_ofs, &alloclen);
if (ret) {
printk(KERN_WARNING "jffs2_reserve_space_gc of
%zd bytes for
garbage_collect_dnode failed: %d\n",
sizeof(ri)+ JFFS2_MIN_DATA_LEN,
ret);
break;
}
cdatalen = min_t(uint32_t, alloclen - sizeof(ri),
end - offset);
datalen = end - offset;
// This looks to be wrong.
writebuf = pg_ptr + (offset & (PAGE_CACHE_SIZE -1));
// I think it should be.
writebuf = pg_ptr + ((offset -start) &
(PAGE_CACHE_SIZE -1));
----- Original Message ----- From: "Alf Nilsson" <alf.nilsson@abem.se>
To: "eCos Disuss" <ecos-discuss@ecos.sourceware.org>
Sent: Tuesday, October 05, 2004 8:02 AM
Subject: [ECOS] JFFS2 fragment corruption during gc?





Hi all,

We've noticed some odd behaviors from the JFFS2 file system and would
like to know if anyone else has had the same type of problems.
Our problem is that when we fill up the file system until it returns
ENOSPC and then start to remove files, the remaining files seem to be
corrupted.
The files are the correct size, but it seems like some fragments are
cleared and set to zero within the file.
We've run tests with the D2 option enabled, but this generates a rather
large amount of  debug info, but I can make is accessible in case anyone
is interested.

I have attached a source code file which causes the error. Depending on
which configuration that is used it takes a different amount of time
before it fails.
In my current repository the fileio2.c file resides in my
fs/jffs2/current/tests directory, with a slightly altered cdl. In case
It would be preferred to send this test as a patch instead let me know.

To make the test run without failing lower the value of FILE_LIMIT, it
seems like if the file system isn't filled up completely the corruption
does not occur.

We have noticed this behavior on 2 different flash circuits and on the
synthetic target.
Our setup has been without compression and CMODE set to NONE default.

The test is based upon writing files that are 75% of the block size, and
filling the file system completely at least 10 times.

I've also attached my exported settings from the ecos.ecc file, in case
anyone want to view which settings I've been using.

Thanks,
Alf Nilsson






----------------------------------------------------------------------------
----




cdl_savefile_version 1;
cdl_savefile_command cdl_savefile_version {};
cdl_savefile_command cdl_savefile_command {};
cdl_savefile_command cdl_configuration { description hardware template


package };


cdl_savefile_command cdl_package { value_source user_value wizard_value


inferred_value };


cdl_savefile_command cdl_component { value_source user_value wizard_value


inferred_value };


cdl_savefile_command cdl_option { value_source user_value wizard_value


inferred_value };


cdl_savefile_command cdl_interface { value_source user_value wizard_value


inferred_value };


cdl_configuration eCos {
   description "" ;
   hardware    linux ;
   template    default ;
   package -hardware CYGPKG_HAL_SYNTH current ;
   package -hardware CYGPKG_HAL_SYNTH_I386 current ;
   package -hardware CYGPKG_DEVS_FLASH_SYNTH current ;
   package -hardware CYGPKG_DEVS_ETH_ECOSYNTH current ;
   package -hardware CYGPKG_DEVS_WATCHDOG_SYNTH current ;
   package -template CYGPKG_HAL current ;
   package -template CYGPKG_IO current ;
   package -template CYGPKG_IO_SERIAL current ;
   package -template CYGPKG_INFRA current ;
   package -template CYGPKG_KERNEL current ;
   package -template CYGPKG_MEMALLOC current ;
   package -template CYGPKG_ISOINFRA current ;
   package -template CYGPKG_LIBC current ;
   package -template CYGPKG_LIBC_I18N current ;
   package -template CYGPKG_LIBC_SETJMP current ;
   package -template CYGPKG_LIBC_SIGNALS current ;
   package -template CYGPKG_LIBC_STARTUP current ;
   package -template CYGPKG_LIBC_STDIO current ;
   package -template CYGPKG_LIBC_STDLIB current ;
   package -template CYGPKG_LIBC_STRING current ;
   package -template CYGPKG_LIBC_TIME current ;
   package -template CYGPKG_LIBM current ;
   package -template CYGPKG_IO_WALLCLOCK current ;
   package -template CYGPKG_ERROR current ;
   package CYGPKG_FS_JFFS2 current ;
   package CYGPKG_IO_FILEIO current ;
   package CYGPKG_IO_FLASH current ;
   package CYGPKG_LINUX_COMPAT current ;
   package CYGPKG_CRC current ;
};

cdl_option CYGDBG_KERNEL_DEBUG_GDB_THREAD_SUPPORT {
   inferred_value 0
};

cdl_option CYGBLD_ISO_CTYPE_HEADER {
   inferred_value 1 <cyg/libc/i18n/ctype.inl>
};

cdl_option CYGBLD_ISO_ERRNO_CODES_HEADER {
   inferred_value 1 <cyg/error/codes.h>
};

cdl_option CYGBLD_ISO_ERRNO_HEADER {
   inferred_value 1 <cyg/error/errno.h>
};

cdl_option CYGBLD_ISO_STDIO_FILETYPES_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_STREAMS_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_FILEOPS_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_FILEACCESS_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_FORMATTED_IO_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_CHAR_IO_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_DIRECT_IO_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_FILEPOS_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDIO_ERROR_HEADER {
   inferred_value 1 <cyg/libc/stdio/stdio.h>
};

cdl_option CYGBLD_ISO_STDLIB_STRCONV_HEADER {
   inferred_value 1 <cyg/libc/stdlib/atox.inl>
};

cdl_option CYGBLD_ISO_STDLIB_ABS_HEADER {
   inferred_value 1 <cyg/libc/stdlib/abs.inl>
};

cdl_option CYGBLD_ISO_STDLIB_DIV_HEADER {
   inferred_value 1 <cyg/libc/stdlib/div.inl>
};

cdl_option CYGBLD_ISO_STRERROR_HEADER {
   inferred_value 1 <cyg/error/strerror.h>
};

cdl_option CYGBLD_ISO_STRTOK_R_HEADER {
   inferred_value 1 <cyg/libc/string/string.h>
};

cdl_option CYGBLD_ISO_STRING_LOCALE_FUNCS_HEADER {
   inferred_value 1 <cyg/libc/string/string.h>
};

cdl_option CYGBLD_ISO_STRING_BSD_FUNCS_HEADER {
   inferred_value 1 <cyg/libc/string/bsdstring.h>
};

cdl_option CYGBLD_ISO_STRING_MEMFUNCS_HEADER {
   inferred_value 1 <cyg/libc/string/string.h>
};

cdl_option CYGBLD_ISO_STRING_STRFUNCS_HEADER {
   inferred_value 1 <cyg/libc/string/string.h>
};

cdl_option CYGBLD_ISO_C_TIME_TYPES_HEADER {
   inferred_value 1 <cyg/libc/time/time.h>
};

cdl_option CYGBLD_ISO_C_CLOCK_FUNCS_HEADER {
   inferred_value 1 <cyg/libc/time/time.h>
};

cdl_option CYGBLD_ISO_SIGNAL_NUMBERS_HEADER {
   inferred_value 1 <cyg/libc/signals/signal.h>
};

cdl_option CYGBLD_ISO_SIGNAL_IMPL_HEADER {
   inferred_value 1 <cyg/libc/signals/signal.h>
};

cdl_option CYGBLD_ISO_SETJMP_HEADER {
   inferred_value 1 <cyg/libc/setjmp/setjmp.h>
};

cdl_option CYGBLD_ISO_DIRENT_HEADER {
   inferred_value 1 <cyg/fileio/dirent.h>
};

cdl_option CYGBLD_ISO_OPEN_MAX_HEADER {
   inferred_value 1 <cyg/fileio/limits.h>
};

cdl_option CYGBLD_ISO_NAME_MAX_HEADER {
   inferred_value 1 <cyg/fileio/limits.h>
};

cdl_option CYGNUM_LIBC_MAIN_DEFAULT_STACK_SIZE {
   inferred_value 16384
};

cdl_option CYGOPT_FS_JFFS2_COMPRESS_ZLIB {
   inferred_value 0
};

cdl_option CYGOPT_FS_JFFS2_COMPRESS_CMODE {
   user_value NONE
};

cdl_component CYGPKG_IO_FILEIO_INODE {
   inferred_value 1
};

cdl_component CYGPKG_IO_FLASH_BLOCK_DEVICE {
   inferred_value 1
};

cdl_option CYGNUM_IO_FLASH_BLOCK_OFFSET_1 {
   user_value 0x00000000
};

cdl_option CYGNUM_IO_FLASH_BLOCK_LENGTH_1 {
   user_value 0x00080000
};

cdl_option CYGNUM_FLASH_SYNTH_BLOCKSIZE {
   user_value 8192
};

cdl_option CYGNUM_FLASH_SYNTH_NUMBLOCKS {
   user_value 1024
};







----------------------------------------------------------------------------
----




#include <pkgconf/hal.h>
#include <pkgconf/kernel.h>
#include <pkgconf/io_fileio.h>
#include <pkgconf/io_flash.h>

#include <cyg/kernel/ktypes.h>         // base kernel types
#include <cyg/infra/cyg_trac.h>        // tracing macros
#include <cyg/infra/cyg_ass.h>         // assertion macros
#include <cyg/io/flash.h>
#include <cyg/io/io.h>
#include <cyg/io/config_keys.h>

#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <errno.h>
#include <string.h>
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>

#include <cyg/fileio/fileio.h>

#include <cyg/infra/testcase.h>
#include <cyg/infra/diag.h>            // HAL polled output

#include <pkgconf/fs_jffs2.h> // Address of JFFS2




//==========================================================================


#define FILE_LIMIT (200)
#define ERROR 1
#define SUCCESS 0




//==========================================================================


#define SHOW_RESULT( _fn, _res ) \
{snprintf( cyg_test_text, sizeof( cyg_test_text ), \
#_fn "() returned %d %s",  (int)_res, _res<0?strerror(errno):""); \
CYG_TEST_INFO( cyg_test_text );}





//==========================================================================


static const char *test_file_folder = "/test_files";
static char *dest;
static char *pattern_buffer;
static int current_file_num = 1;
static int oldest_file_num = 1;
static int file_size;
static char cyg_test_text[200];
static int out_of_space = 0;




//==========================================================================


static void create_test_pattern( char *buffer, size_t s );
static void create_filename( char*buffer, size_t size, int i );
static int verify_files( const char *path,size_t size );
static void remove_file( int i );
static int need_space( void );




//==========================================================================


static void create_test_pattern( char *buffer, size_t s )
{
   memset( buffer, '8', s );
}




//==========================================================================


static void create_filename( char*buffer, size_t size, int i )
{
   snprintf( buffer, size, "%s/%i", test_file_folder, i );
}




//==========================================================================


static int verify_files( const char *path, size_t size)
{
   DIR *dirp;
   int fd;
   struct dirent *entry;
   ssize_t read_size;
   int error = 0;
   char absolute_filename[50];
   int err;

   dirp = opendir( path );
   if( dirp )
   {
       entry = readdir( dirp );
while( entry )
{
   if( strcmp( entry->d_name, "." ) != 0 &&
strcmp( entry->d_name, ".." ) != 0 )
   {
     create_filename( absolute_filename,
      sizeof( absolute_filename ),
      atoi( entry->d_name ));

fd = open( absolute_filename, O_RDONLY );
if( fd != -1 )
{
   int wrong = 0;
   read_size = read( fd, dest, size );
   if( read_size != -1 )
   {
       int j;
for(j=0;j<read_size;j++)
{
   if( pattern_buffer[j] != dest[j] )
       wrong++;
}
if( wrong )
   error = 1;
if( !( read_size == size || read_size==0 ) )
       {
   snprintf( cyg_test_text, sizeof( cyg_test_text ), "Bad size %s %d",
     entry->d_name, (int)read_size );
   CYG_TEST_INFO( cyg_test_text );

}
   }
   else
   {
       if( errno == ENOSPC ) out_of_space = 1;
       SHOW_RESULT( read, read_size );
error = 1;
   }

   err = close( fd );
   if( err == -1 )
   {
       if( errno == ENOSPC ) out_of_space = 1;
       SHOW_RESULT( close, err );
   }
   if( error )
   {
       snprintf( cyg_test_text, sizeof( cyg_test_text ),
 "Compare of file : %s failed", entry->d_name );
CYG_TEST_INFO( cyg_test_text );
snprintf( cyg_test_text, sizeof( cyg_test_text ),
 "Total %d of %d bytes differs", wrong, (int)read_size );
CYG_TEST_INFO( cyg_test_text );
break;
   }
}
else
{
   if( errno == ENOSPC ) out_of_space = 1;
   SHOW_RESULT( open, fd );
   err = 1;
   break;
}
   }
   entry = readdir( dirp );
}
err = closedir( dirp );
if( err == -1 )
{
   if( errno == ENOSPC ) out_of_space = 1;
   SHOW_RESULT( closedir, err );
}
   }

   if( error )
       return ERROR;

   return SUCCESS;
}




//==========================================================================


static void remove_file( int i )
{
   int err;
   char filenamebuff[50];
   create_filename(filenamebuff,
   sizeof(filenamebuff),
   i );
   snprintf( cyg_test_text, sizeof( cyg_test_text ),
     "Remove: %s", filenamebuff );
   CYG_TEST_INFO( cyg_test_text );
   memset( cyg_test_text, 0, sizeof( cyg_test_text ) );

   err = unlink( filenamebuff );
   if( err == -1 )
   {
       if( errno == ENOSPC ) out_of_space = 1;
       SHOW_RESULT( unlink, err );
   }

   if( verify_files( test_file_folder, file_size ) != SUCCESS )
       CYG_TEST_FAIL_FINISH("File verify failed");
}




//==========================================================================


static int create_file( int i, const char *data, size_t count )
{
   int fd;
   int err = 0;
   int err_check;
   char filename[50];

create_filename( filename, sizeof( filename ), i );

   fd = open( filename, O_WRONLY | O_CREAT );
   if (fd==-1)
   {
       if( errno == ENOSPC ) out_of_space = 1;
       SHOW_RESULT( open, fd );
if( verify_files( test_file_folder, count ) != SUCCESS )
   CYG_TEST_FAIL_FINISH("File verify failed");
return -1;
   }

   snprintf( cyg_test_text, sizeof( cyg_test_text ),
     "Created file %s", filename );
   CYG_TEST_INFO( cyg_test_text );
   memset( cyg_test_text, 0, sizeof( cyg_test_text ) );

   if( verify_files( test_file_folder, count ) != SUCCESS)
       CYG_TEST_FAIL_FINISH("File verify failed");

   if( fd != -1 )
   {
       err = write( fd, data, count );
if( err != count )
       {
   if( errno == ENOSPC ) out_of_space = 1;
   SHOW_RESULT( write, err );
   err = -1;
}

err_check = close( fd );
if( err_check == -1 )
{
   if( errno == ENOSPC ) out_of_space = 1;
   SHOW_RESULT( close, err_check );
}

if( err != count )
   remove_file( i );
   }

   if( verify_files( test_file_folder, count ) != SUCCESS )
       CYG_TEST_FAIL_FINISH("File verify failed");

   return err;
}




//==========================================================================


static int need_space( void )
{
   if( out_of_space )
   {
       out_of_space = 0;
return true;
   }
   if( ( current_file_num - oldest_file_num ) >= FILE_LIMIT )
       return true;

   return false;
}




//==========================================================================


// main

int main( int argc, char **argv )
{
   int err;
   Cyg_ErrNo cyg_err;
   cyg_io_handle_t flash_handle;
   cyg_io_flash_getconfig_blocksize_t blocksize;
   size_t len;
   int max_num_of_files;

CYG_TEST_INIT();

// --------------------------------------------------------------

   cyg_err = cyg_io_lookup( CYGDAT_IO_FLASH_BLOCK_DEVICE_NAME_1,
    &flash_handle );
   if( cyg_err != ENOERR )
   {
       CYG_TEST_FAIL_FINISH( "Could not read flash device" );
   }

   len = sizeof(blocksize);
   memset( &blocksize, 0, len );
   cyg_err = cyg_io_get_config( flash_handle,
    CYG_IO_GET_CONFIG_FLASH_BLOCKSIZE, &blocksize, &len);
   if( cyg_err != ENOERR )
   {
       CYG_TEST_FAIL_FINISH("Could not read flash blocksize" );
   }

file_size = ( blocksize.block_size / 4 ) * 3;

max_num_of_files = ( CYGNUM_IO_FLASH_BLOCK_LENGTH_1 / file_size ) *


10;


   snprintf( cyg_test_text, sizeof( cyg_test_text ),
     "File size = %d, Max files = %d", file_size, max_num_of_files );
   CYG_TEST_INFO( cyg_test_text );

   dest = malloc( file_size );
   pattern_buffer = malloc( file_size );

// --------------------------------------------------------------

   err = mount( CYGDAT_IO_FLASH_BLOCK_DEVICE_NAME_1, "/", "jffs2" );
   if( err < 0 ) SHOW_RESULT( mount, err );

   err = chdir( "/" );
   if( err < 0 ) SHOW_RESULT( chdir, err );

   err = mkdir( test_file_folder, 0 );
   if( err < 0 ) SHOW_RESULT( mkdir, err );

// --------------------------------------------------------------

   // Init buffer
   create_test_pattern( pattern_buffer, file_size );

   while( max_num_of_files-- )
   {
       int first_file_num = current_file_num++;

while( create_file( first_file_num, pattern_buffer, file_size ) == -1 )
{
   if( need_space() )
       remove_file( oldest_file_num++ );
}
if( need_space() )
{
   remove_file( oldest_file_num++ );
}
   }

   free( dest );
   free( pattern_buffer );

   if( max_num_of_files > 0 )
   {
     snprintf( cyg_test_text, sizeof( cyg_test_text ),
"File count was not reached %d files left", max_num_of_files );
     CYG_TEST_FAIL_FINISH( cyg_test_text );
   }

   err = chdir( "/" );
   if( err < 0 ) SHOW_RESULT( chdir, err );

   err = umount( "/" );
   if( err < 0 ) SHOW_RESULT( umount, err );

   CYG_TEST_PASS_FINISH("fileio2");
}






----------------------------------------------------------------------------
----




--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss







--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]