“In the enter critical section function called in a task, the first instruction executed is a DISABLE interrupts. In our case, a higher priority interrupt occured just right after the disable interrupt instruction. In the interrupt handler, an enter and exit critical section was called. In the exit critical section from the handler, the interrupts were enabled again because the nesting counter was set to 1. ”
I have not read past this paragraph but have already noted two things that are wrong with your application and guess the rest of your post relates to that.
1) You must not use taskENTER_CRITICAL from an ISR. In fact, you must not call any API functions that don't end in FromISR in an ISR. You can use
uxSavedInterruptStatus = portSET_INTERRUPT_MASK_FROM_ISR();
portCLEAR_INTERRUPT_MASK_FROM_ISR( uxSavedInterruptStatus );
(code cut from queue.c)
2) If you have an interrupt calling API function when the task is in a critical section then the rule that interrupts that use the FromISR API functions must have a priority at or below configMAX_SYSCALL_INTERRUPT_PRIORITY.
Some links that might help you
http://www.freertos.org/FAQHelp.html
http://www.freertos.org/RTOS-Cortex-M3-M4.html