Taking folks' suggestions, here's the first iteration:
It's still a bit heavy memory wise, but a lot better. I also think it's a bit cleaner and am experimenting with just using grid as a player frame replacement.
Also, I no longer can see my cooldowns with all action bars hidden. Someone mentioned CooldownCount2, but googling that didn't turn up anywhere to download it.
edit: As Kalman points out, the mod in question is called CooldownTimers2.
Addon list:

!BugGrabber
!Swatter
Ace2
Aloft
ArenaPointer
ArkInventory
Assessment
Auc-Advanced
Auc-Filter-Basic
Auc-ScanData
Auc-Stat-Purchased
Auc-Stat-Simple
Auc-Stat-StdDev
AuldLangSyne
AuldLangSyne_Backup
AuldLangSyne_FriendList
AuldLangSyne_Fu
AuldLangSyne_GuildList
AuldLangSyne_Info
AuldLangSyne_Note
AuldLangSyne_Panel
AuldLangSyne_Sync
AutoBar
AutoBarConfig
Automaton
AutoTurnin
Bartender3
BigBrother
Blizzard_AuctionUI
Blizzard_BattlefieldMinimap
Blizzard_BindingUI
Blizzard_CombatText
Blizzard_CraftUI
Blizzard_GMSurveyUI
Blizzard_InspectUI
Blizzard_ItemSocketingUI
Blizzard_MacroUI
Blizzard_RaidUI
Blizzard_TalentUI
Blizzard_TradeSkillUI
Blizzard_TrainerUI
BtmScan
Buffalo2
BugSack
BulkMail2
BulkMail2Inbox
Capping
CECB_Debuffs
CECB_Options
CECB_PvEModule
CECB_PvPModule
CEnemyCastBar
CooldownCount
DynRepBar
Enchantrix
Enchantrix-Barker
EnhTooltip
EQCompare
ErrorMonster
FreeRefills
FuBar
FuBar_ClockFu
FuBar_CraftTimersFu
FuBar_PerformanceFu
FuBar_QuestsFu
Grid
GridManaBars
GridStatusMissingBuffs
HealPoints
Informant
ItemRack
ItemRackFu
LibBabble-Spell-3.0
LightHeaded
LightHeaded_A20_Data
LightHeaded_A40_Data
LightHeaded_A60_Data
LightHeaded_A80_Data
LightHeaded_deDE_Data
LightHeaded_H20_Data
LightHeaded_H40_Data
LightHeaded_H60_Data
LightHeaded_H80_Data
LightHeaded_NPC_Data
LightHeaded_Quest_Data
MobHealth
Omen
Parrot
PitBull
PitBull_AltAura
PitBull_Aura
PitBull_Banzai
PitBull_BarFader
PitBull_CastBar
PitBull_CombatFader
PitBull_CombatIcon
PitBull_CombatText
PitBull_ComboPoints
PitBull_DruidManaBar
PitBull_EstimatedHeals
PitBull_ExperienceBar
PitBull_HealthBar
PitBull_HideBlizzard
PitBull_Highlight
PitBull_InteractLines
PitBull_LeaderIcon
PitBull_MasterLooterIcon
PitBull_Portrait
PitBull_PowerBar
PitBull_PvPIcon
PitBull_RaidTargetIcon
PitBull_RangeCheck
PitBull_ReputationBar
PitBull_RestIcon
PitBull_Spark
PitBull_ThreatBar
PitBull_VoiceIcon
Prat
Prat_AutoLoD
Prat_Modules
Prat_Modules_AutoLoD
Proximo
Quartz
Quartz_Buff
Quartz_Flight
Quartz_Focus
Quartz_GCD
Quartz_Interrupt
Quartz_Latency
Quartz_Mirror
Quartz_Player
Quartz_Range
Quartz_Swing
Quartz_Target
Quartz_Timer
Quartz_Tradeskill
RatingBuster
SendSelf
Squeenix
Stubby
Talented
Talented_Data
TinyTip
TinyTipExtras
TinyTipOptions
TomTom
TooltipExchange
TooltipExchange_ItemDatabase
TourGuide
VendorBait
Voyeur